-<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
- <title xml:id="configuringlnet.title">Configuring Lustre Networking (LNET)</title>
+<?xml version='1.0' encoding='UTF-8'?>
+<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
+ <title xml:id="configuringlnet.title">Configuring Lustre Networking (LNET) </title>
<para>This chapter describes how to configure Lustre Networking (LNET). It includes the following sections:</para>
<itemizedlist>
<listitem>
</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438216_35668"/>
- </para>
- </listitem>
- <listitem>
<para><xref linkend="dbdoclet.50438216_15200"/>
</para>
</listitem>
</itemizedlist>
<note>
- <para>Configuring LNET is optional.</para>
- <para>LNET will, by default, use the first TCP/IP interface it discovers on a system
- (<literal>eth0</literal>). If this network configuration is sufficient, you do not need to
- configure LNET. LNET configuration is required if you are using InfiniBand or multiple
- Ethernet interfaces.</para>
+ <para>LNET will, by default, use the first TCP/IP interface it discovers on a system. If the
+ default is sufficient, further configuration is not required.</para>
</note>
<section xml:id="dbdoclet.50438216_33148">
- <title><indexterm><primary>LNET</primary></indexterm>
-
- Overview of LNET Module Parameters</title>
- <para>LNET kernel module (lnet) parameters specify how LNET is to be configured to work with the
- Lustre file system, including which NICs will be configured to work with Lustre file system
- and the routing to be used with Lustre file system.</para>
- <para>Parameters for LNET can be specified in the <literal>/etc/modprobe.d/lustre.conf</literal>
- file. In some cases the parameters may have been stored in
- <literal>/etc/modprobe.conf</literal>, but this has been deprecated since before Red Hat
- Enterprise Linux 5 and SUSE Linux Enterprise Server 10 were released. A separate
+ <title><indexterm><primary>LNet</primary></indexterm>
+ Overview of <literal>lnet</literal> Module Parameters</title>
+ <para>The <literal>lnet</literal> kernel module parameters specify how LNET is to be configured
+ to work with a Lustre file system, including details on which NICs are to be configured and
+ how routing among LNET routers will take place.</para>
+ <para>Parameters for <literal>lnet</literal> can be specified in the
+ <literal>/etc/modprobe.d/lustre.conf</literal> file. The file need not be named
+ <literal>lustre.conf</literal>. However, this convention will be used in this manual. For
+ distributions previous to Red Hat Enterprise Edition 5 and SUSE Linux Enterprise Server 10,
+ <literal>lnet</literal> parameters may have been stored in
+ <literal>/etc/modprobe.conf</literal>, but this method of specifying <literal>lnet</literal>
+ parameters is now deprecated. Storing LNET parameters in a separate
<literal>/etc/modprobe.d/lustre.conf</literal> file simplifies administration and
distribution of the Lustre networking configuration. This file contains one or more entries
with the syntax:</para>
<screen>options lnet <replaceable>parameter</replaceable>=<replaceable>value</replaceable></screen>
- <para>To specify the network interfaces that are to be used for Lustre, set either the <literal>networks</literal> parameter or the <literal>ip2nets</literal> parameter (only one of these parameters can be used at a time):</para>
+ <para>To specify the network interfaces that are to be used for the Lustre file system, set
+ either the <literal>networks</literal> parameter or the <literal>ip2nets</literal> parameter
+ (only one of these parameters can be used at a time):</para>
<itemizedlist>
<listitem>
- <para><literal>networks</literal> - Specifies the networks to be used.</para>
+ <para><literal>networks</literal> - Specifies the LNET network interfaces to be used.</para>
</listitem>
<listitem>
- <para><literal>ip2nets</literal> - Lists globally-available networks, each with a range of IP addresses. LNET then identifies locally-available networks through address list-matching lookup.</para>
+ <para><literal>ip2nets</literal> - Lists globally-available networks, each with a range of
+ IP addresses. LNET then identifies locally-available networks through address
+ list-matching lookup.</para>
</listitem>
</itemizedlist>
<para>See <xref linkend="dbdoclet.50438216_46279"/> and <xref linkend="dbdoclet.50438216_31414"/> for more details.</para>
- <para>To set up routing between networks, use:</para>
- <itemizedlist>
- <listitem>
- <para><literal>routes</literal> - Lists networks and the NIDs of routers that forward to them.</para>
- </listitem>
- </itemizedlist>
- <para>See <xref linkend="dbdoclet.50438216_71227"/> for more details.</para>
- <para>A <literal>router</literal> checker can be configured to enable Lustre nodes to detect router health status, avoid routers that appear dead, and reuse those that restore service after failures. See <xref linkend="dbdoclet.50438216_35668"/> for more details.</para>
- <para>For a complete reference to the LNET module parameters, see <emphasis><xref linkend="configurationfilesmoduleparameters"/>LNET Options</emphasis>.</para>
- <note>
- <para>We recommend that you use 'dotted-quad' notation for IP addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
- </note>
- <section remap="h3">
- <title><indexterm><primary>LNET</primary><secondary>using NID</secondary></indexterm>Using a Lustre Network Identifier (NID) to Identify a Node</title>
- <para>A Lustre network identifier (NID) is used to uniquely identify a Lustre network endpoint by node ID and network type. The format of the NID is:</para>
- <screen><replaceable>network_id</replaceable>@<replaceable>network_type</replaceable></screen>
- <para>Examples are:</para>
- <screen>10.67.73.200@tcp0
-10.67.75.100@o2ib</screen>
- <para>The first entry above identifies a TCP/IP node, while the second entry identifies an InfiniBand node.</para>
- <para>When a mount command is run on a client, the client uses the NID of the MDS to retrieve configuration information. If an MDS has more than one NID, the client should use the appropriate NID for its local network.</para>
- <para>To determine the appropriate NID to specify in the mount command, use the <literal>lctl</literal> command. To display MDS NIDs, run on the MDS :</para>
- <screen>lctl list_nids</screen>
- <para>To determine if a client can reach the MDS using a particular NID, run on the client:</para>
- <screen>lctl which_nid <replaceable>MDS_NID</replaceable></screen>
+ <para>To set up routing between networks, use the <literal>routes</literal> parameter to
+ identify the LNET subnets and the NID(s) of the routers that forward to them. See <xref
+ linkend="dbdoclet.50438216_71227"/> for more details.</para>
</section>
- </section>
<section xml:id="dbdoclet.50438216_46279">
- <title><indexterm><primary>LNET</primary><secondary>module parameters</secondary></indexterm>Setting the LNET Module networks Parameter</title>
- <para>If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the <literal>lustre.conf</literal> file on the node that sets the LNET module <literal>networks</literal> parameter:</para>
- <screen>options lnet networks=<replaceable>comma-separated list of networks</replaceable></screen>
- <para>This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface:</para>
- <screen>options lnet networks=tcp0(eth0),o2ib(ib0)</screen>
- <para>This example specifies that the Lustre node will use the TCP/IP interface <literal>eth1</literal>:</para>
+ <title><indexterm><primary>LNet</primary><secondary>module parameters</secondary></indexterm><literal>networks</literal> Parameter</title>
+ <para>The <literal>networks</literal> parameter maps a network interface to an LNET subnet. You
+ can do this by including an entry or a comma separated list in the
+ <literal>lustre.conf</literal> file.</para>
+ <screen>options lnet networks=<replaceable>LNET(interface),LNET(interface),...</replaceable></screen>
+ <para>For example:</para>
+ <para>To map the LNET <literal>tcp0</literal> to the Ethernet interface
+ <literal>eth1</literal>:</para>
<screen>options lnet networks=tcp0(eth1)</screen>
<para>Depending on the network design, it may be necessary to specify explicit interfaces. To explicitly specify that interface <literal>eth2</literal> be used for network <literal>tcp0</literal> and <literal>eth3</literal> be used for <literal>tcp1</literal> , use this entry:</para>
<screen>options lnet networks=tcp0(eth2),tcp1(eth3)</screen>
<para>LNET lines in <literal>lustre.conf</literal> are only used by the local node to determine what to call its interfaces. They are not used for routing decisions.</para>
</note>
<section remap="h3">
- <title><indexterm><primary>configuring</primary><secondary>multihome</secondary></indexterm>Multihome Server Example</title>
- <para>If a server with multiple IP addresses (multihome server) is connected to a Lustre network, certain configuration setting are required. An example illustrating these setting consists of a network with the following nodes:</para>
+ <title><indexterm><primary>configuring</primary><secondary>multi-homed</secondary></indexterm>Multi-homed Server Example</title>
+ <para>If a server with multiple IP addresses (multi-homed server) is connected to a Lustre network, certain configuration settings are required. An example illustrating these setting consists of a network with the following nodes:</para>
<itemizedlist>
<listitem>
- <para> Server svr1 with three TCP NICs (<literal>eth0</literal>, <literal>eth1</literal>, and <literal>eth2</literal>) and an InfiniBand NIC.</para>
+ <para> Server <literal>svr1</literal> with three Ethernet NIC s (<literal>eth0</literal>,
+ <literal>eth1</literal>, and <literal>eth2</literal>) and an InfiniBand NIC.</para>
</listitem>
<listitem>
- <para> Server svr2 with three TCP NICs (<literal>eth0</literal>, <literal>eth1</literal>, and <literal>eth2</literal>) and an InfiniBand NIC. Interface eth2 will not be used for Lustre networking.</para>
+ <para> Server <literal>svr2</literal> with three Ethernet NICs (<literal>eth0</literal>,
+ <literal>eth1</literal>, and <literal>eth2</literal>) and an InfiniBand NIC. The
+ interface <literal>eth2</literal> will not be used for Lustre networking.</para>
</listitem>
<listitem>
- <para> TCP clients, each with a single TCP interface.</para>
+ <para> TCP clients, each with a single Ethernet interface.</para>
</listitem>
<listitem>
- <para> InfiniBand clients, each with a single InfiniBand interface and a TCP/IP interface
- for administration.</para>
+ <para> InfiniBand clients, each with a single InfiniBand interface and an Ethernet
+ interface for administration.</para>
</listitem>
</itemizedlist>
<para>To set the <literal>networks</literal> option for this example:</para>
<para> On each server, <literal>svr1</literal> and <literal>svr2</literal>, include the following line in the <literal>lustre.conf</literal> file:</para>
</listitem>
</itemizedlist>
- <screen>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib</screen>
+ <screen>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib(ib0)</screen>
<itemizedlist>
<listitem>
<para> For TCP-only clients, the first available non-loopback IP interface is used for <literal>tcp0</literal>. Thus, TCP clients with only one interface do not need to have options defined in the <literal>lustre.conf</literal> file.</para>
<para> On the InfiniBand clients, include the following line in the <literal>lustre.conf</literal> file:</para>
</listitem>
</itemizedlist>
- <screen>options lnet networks=o2ib</screen>
- <note>
- <para>By default, Lustre ignores the loopback (<literal>lo0</literal>) interface. Lustre does not ignore IP addresses aliased to the loopback. If you alias IP addresses to the loopback interface, you must specify all Lustre networks using the LNET networks parameter.</para>
- </note>
- <note>
- <para>If the server has multiple interfaces on the same subnet, the Linux kernel will send
- all traffic using the first configured interface. This is a limitation of the Linux
- kernel, not the Lustre software. In this case, network interface bonding should be used.
- For more information about network interface bonding, see <xref linkend="settingupbonding"
- />.</para>
- </note>
- </section>
+ <screen>options lnet networks=o2ib(ib0)</screen>
+ <para>If a node has only one InfiniBand interface, then the interface name need not be
+ specified. <literal>- ib0</literal> will be assumed.</para>
+ </section>
</section>
<section xml:id="dbdoclet.50438216_31414">
- <title><indexterm><primary>LNET</primary><secondary>ip2nets</secondary></indexterm>Setting the LNET Module ip2nets Parameter</title>
+ <title><indexterm><primary>LNet</primary><secondary>ip2nets</secondary></indexterm><literal>ip2nets</literal> Parameter</title>
<para>The <literal>ip2nets</literal> option is typically used when a single, universal <literal>lustre.conf</literal> file is run on all servers and clients. Each node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses.</para>
- <para>Note that the IP address patterns listed in the <literal>ip2nets</literal> option are <emphasis>only</emphasis> used to identify the networks that an individual node should instantiate. They are <emphasis>not</emphasis> used by LNET for any other communications purpose.</para>
+ <para>Note that the IP address patterns listed in the <literal>ip2nets</literal> option are
+ <emphasis>only</emphasis> used to identify the networks that an individual node should
+ instantiate. They are <emphasis>not</emphasis> used by LNET for any other communications
+ purpose.</para>
<para>For the example below, the nodes in the network have these IP addresses:</para>
<itemizedlist>
<listitem>
tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'</screen>
<para>Each entry in <literal>ip2nets</literal> is referred to as a 'rule'.</para>
<para>The order of LNET entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in <literal>lustre.conf</literal> will be used.</para>
- <para>Because <literal>svr1</literal> and <literal>svr2</literal> match the first rule, LNET uses <literal>eth0</literal> for <literal>tcp0</literal> on those machines. (Although <literal>svr1</literal> and <literal>svr2</literal> also match the second rule, the first matching rule for a particular network is used).</para>
- <para>The <literal>[2-8/2]</literal> format indicates a range of 2-8 stepped by 2; that is 2,4,6,8. Thus, the clients at <literal>132.6.3.5</literal> will not find a matching o2ib network.</para>
+ <para>Because <literal>svr1</literal> and <literal>svr2</literal> match the first rule, LNET
+ uses <literal>eth0</literal> for <literal>tcp0</literal> on those machines. (Although
+ <literal>svr1</literal> and <literal>svr2</literal> also match the second rule, the first
+ matching rule for a particular network is used).</para>
+ <para>The <literal>[2-8/2]</literal> format indicates a range of 2-8 stepped by 2; that is
+ 2,4,6,8. Thus, the clients at <literal>132.6.3.5</literal> will not find a matching
+ <literal>o2ib</literal> network.</para>
</section>
<section xml:id="dbdoclet.50438216_71227">
- <title><indexterm><primary>LNET</primary><secondary>routes</secondary></indexterm>Setting the LNET Module routes Parameter</title>
- <para>The LNET module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in <literal>modprobe.conf</literal> on each Lustre node.</para>
- <para>The LNET routes parameter specifies a colon-separated list of router definitions. Each route is defined as a network number, followed by a list of routers:</para>
- <screen>routes=<replaceable>net_type router_NID(s)</replaceable></screen>
- <para>This example specifies bi-directional routing in which TCP clients can reach Lustre resources on the IB networks and IB servers can access the TCP networks:</para>
- <screen>options lnet 'ip2nets="tcp0 192.168.0.*; \
- o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \
- o2ib0 192.16.8.0.[1-8]@tcp0"'</screen>
- <para>All LNET routers that bridge two networks are equivalent. They are not configured as primary or secondary, and the load is balanced across all available routers.</para>
- <para>The number of LNET routers is not limited. Enough routers should be used to handle the required file serving bandwidth plus a 25 percent margin for headroom.</para>
- <section remap="h3">
- <title><indexterm><primary>LNET</primary><secondary>routing example</secondary></indexterm>Routing Example</title>
- <para>On the clients, place the following entry in the <literal>lustre.conf</literal> file</para>
- <screen>lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0"</screen>
- <para>On the router nodes, use:</para>
- <screen>lnet networks="tcp o2ib" forwarding=enabled </screen>
- <para>On the MDS, use the reverse as shown below:</para>
- <screen>lnet networks="o2ib0" routes="tcp0 132.6.1.[1-8]@o2ib0" </screen>
- <para>To start the routers, run:</para>
- <screen>modprobe lnet
+ <title><indexterm>
+ <primary>LNet</primary>
+ <secondary>routes</secondary>
+ </indexterm>Using the <literal>routes</literal> Parameter</title>
+ <para>Before discussing the <literal>routes</literal> parameter, it is important to look at what
+ LNET routing achieves and why it is needed.</para>
+ <para>The role of an LNET router is essentially that of a gateway that forwards Lustre traffic
+ from one LNET network to one (or more) other LNET networks. This implies that at the LNET
+ layer, an LNET router is the next hop between two nodes on different LNET networks. Note that
+ this is orthogonal to whether or not the nodes in question are in the same IP subnet or are
+ separated by one or more IP routers. Therefore, LNET routing is not required when clients and
+ servers are all in the same LNET network.</para>
+ <para>However, an LNET router is more than just a regular gateway; it is an LNET intelligent
+ gateway. It is important to remember that Lustre routing is static routing, where the LNET
+ network topology is statically configured and parsed when the system initializes. Routers can
+ die and also come back online during operation. These topics will be discussed in greater
+ detail in the following sections.</para>
+ <section remap="h3"><title><indexterm><primary>LNet></primary><secondary>Routes Parameter</secondary></indexterm><literal>routes</literal> Parameter</title>
+ <para>The <literal>routes</literal> parameter is used to tell a node which route to use when
+ forwarding traffic by identifying LNET routers in a Lustre configuration. The routes
+ parameter needs to be set in the <literal>lustre.conf</literal> file to specify the next hop
+ router.</para>
+ <para>The <literal>routes</literal> parameter specifies a semi-colon separated list of router definitions.</para>
+ <screen>routes=<replaceable>dest_lnet [hop] [priority] router_NID@src_lnet; \
+ dest_lnet [hop] [priority] router_NID@src_lnet</replaceable></screen>
+ <para>An alternative syntax consists of a colon separated list of router definitions:</para>
+ <screen>routes=<replaceable>dest_lnet: [hop] [priority] router_NID@src_lnet \
+ [hop] [priority] router_NID@src_lnet</replaceable></screen>
+ <para>When there are two or more LNET routers, it is possible to give weighted priorities to
+ each router using the <literal>priority</literal> parameter. Some reasons for doing this are
+ that one of the routers is more capable than the other, or one is a primary router and the
+ other is a back up, or one is for one section of clients and the other is for another
+ section, or each router is moving traffic to a different physical location.The
+ <literal>priority</literal> parameter is optional and need not be specified if no priority
+ exists.</para>
+ <para>The <literal>hop</literal> parameter specifies the number of hops to the destination.
+ When a node forwards traffic, the route with the least hops is used. If multiple routes to
+ the same destination network have the same number of hops, the traffic is distributed
+ between these routes in a round-robin fashion. To reach/transmit to the LNET
+ <literal>dest_lnet</literal>, the next hop for a given node is the LNET router with the
+ NID <literal>router_NID</literal> in the LNET <literal>src_lnet</literal>. </para>
+ <para>Given a sufficiently well-architected system, it is possible to map the flow to and from
+ every single client or server. This type of routing has also been called
+ <emphasis>fine-grained</emphasis> routing.</para>
+ </section>
+ <section remap="h3"><title><indexterm><primary>LNet></primary><secondary>Router configurations</secondary></indexterm>Router Configurations</title>
+ <para>To get a router setup started and running, execute the following commands:</para>
+ <screen>modprobe lnet
lctl network configure</screen>
+ <para> OR</para>
+ <screen>lctl network up</screen>
+ <para>For a router to forward traffic, the configuration parameter
+ <literal>forwarding</literal> must be set to <literal>enabled</literal> (it is set to
+ <literal>disabled</literal> by default). If forwarding is not enabled, traffic destined
+ for the node is dropped.</para>
+ <para>
+ <note>
+ <para>Any changes to the <literal>lustre.conf</literal> modprobe file require unloading
+ the <literal>lnet</literal> kernel module and repeating the above given commands. Thus
+ any major changes, especially on the servers, should be made with due
+ consideration.</para>
+ </note>
+ </para>
+ <para>Here are a few examples starting from a basic setup and increasing in complexity to
+ better illustrate LNET routing:</para>
+
+ <itemizedlist>
+ <listitem>
+ <para><emphasis role="italic">A simple setup involving one TCP client, one router and
+ servers(MGS,MDS,OSS) in an InfiniBand fabric.</emphasis></para>
+ <para>The LNET router has two NIDs, 192.168.1.2@tcp0 and 10.13.24.90@o2ib0.</para>
+ <para>The <literal>lustre.conf</literal> file for the client includes:</para>
+ <screen>options lnet networks="tcp0(eth0)" routes="o2ib0 192.168.1.2@tcp0"</screen>
+ <para>On the router nodes:</para>
+ <screen>options lnet networks="o2ib0(ib0),tcp0(eth0)" forwarding=enabled</screen>
+ <para>On the server nodes:</para>
+ <screen>options lnet networks="o2ib0(ib0)" routes="tcp0 10.13.24.90@o2ib0"</screen>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="italic">Adding more routers to above described case.</emphasis></para>
+ <para>When two or more routers are specified in the file, they are considered equivalent and are
+ not configured as primary or secondary. The load is directed to the routers based on the
+ <literal>route</literal> parameter settings for <literal>priority</literal> and
+ <literal>hop</literal> (see <xref xmlns:xlink="http://www.w3.org/1999/xlink"
+ linkend="dbdoclet.50438216_71227"/>). The syntax on the client nodes is:</para>
+ <screen>options lnet networks="tcp0(eth0)" routes="o2ib0 192.168.1.2@tcp0;
+ o2ib0 192.168.1.3@tcp0"</screen>
+ <para>The servers can be configured likewise. The number of LNET routers is not limited. Enough routers should be used to handle the required file serving bandwidth plus a 25 percent margin for headroom. The nodes are capable of recovering if one or more routers fail, provided, there is still at least one router left running. The nodes are also capable of recognizing when an offline routers comes online.</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="italic">Assigning priorities to routers.</emphasis></para>
+ <screen>options lnet networks="tcp0(eth0)" routes="o2ib0 1 192.168.1.2@tcp0;
+ o2ib0 2 192.168.1.3@tcp0"</screen>
+ <para>The above example specifies two routers with priorities 1 and 2. In this case, the node will always first send traffic via 192.168.1.2 and if and only if that router is down, will it send traffic via 192.168.1.3.</para>
+ <para>It is important to note that the weights are pre-configured and do not or cannot be
+ changed dynamically. This method gives more control to an administrator on deciding
+ which is a better route than just the route with least number of hops.</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="italic">Assigning equal priorities to routers.</emphasis></para>
+ <screen>options lnet networks="tcp0(eth0)" routes="o2ib0 1 192.168.1.2@tcp0;
+ o2ib0 1 192.168.1.3@tcp0"</screen>
+ <para>The above example has two routers of the same priority. This is treated the same as
+ specifying no priority at all.</para>
+ </listitem>
+
+ <listitem>
+ <para><emphasis role="italic">Creating classes of routers.</emphasis></para>
+ <screen>options lnet networks="tcp0(eth0)" routes="o2ib0 1 192.168.1.2@tcp0;
+ o2ib0 1 192.168.1.3@tcp0;o2ib0 2 192.168.1.4@tcp0;o2ib0 2 192.168.1.5@tcp0"</screen>
+ <para>This example creates two classes of routers load balanced between themselves. In this case, the node will only use the routers with priority 2 only if both the routers with priority 1 are not running.</para>
+ </listitem>
+ </itemizedlist>
</section>
</section>
<section xml:id="dbdoclet.50438216_10523">
<screen>options lnet <replaceable>router_checker_parameter</replaceable>=<replaceable>value</replaceable></screen>
<para>The router checker parameters are:</para>
<itemizedlist>
- <listitem>
- <para><literal>live_router_check_interval</literal> - Specifies a time interval in seconds after which the router checker will ping the live routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
- <screen>options lnet live_router_check_interval=60</screen>
- </listitem>
- <listitem>
- <para><literal>dead_router_check_interval</literal> - Specifies a time interval in seconds after which the router checker will check for dead routers. The default value is 0, meaning no checking is done. To set the value to 60, enter:</para>
- <screen>options lnet dead_router_check_interval=60</screen>
- </listitem>
- <listitem>
- <para>auto_down - Enables/disables (1/0) the automatic marking of router state as up or down. The default value is 1. To disable router marking, enter:</para>
- <screen>options lnet auto_down=0</screen>
- </listitem>
- <listitem>
- <para><literal>router_ping_timeout</literal> - Specifies a timeout for the router checker when it checks live or dead routers. The router checker sends a ping message to each dead or live router once every dead_router_check_interval or live_router_check_interval respectively. The default value is 50. To set the value to 60, enter:</para>
- <screen>options lnet router_ping_timeout=60</screen>
- <note>
+ <listitem>
+ <para><literal>auto_down</literal> - Enables/disables (1/0) the automatic marking of router state as up or down. The default value is 1. To disable router marking, enter:</para>
+ <screen>options lnet auto_down=0</screen>
+ </listitem>
+ <listitem>
+ <para><literal>avoid_asym_router_failure</literal> - Specifies that if even one interface of a
+ router is down for some reason, the entire router is marked as down. This is important
+ because if nodes are not aware that the interface on one side is down, they will still
+ keep pushing data to the other side presuming that the router is healthy, when it really
+ is not. To turn it on, enter:</para>
+ <screen>options lnet avoid_asym_router_failure=1</screen>
+ </listitem>
+ <listitem>
+ <para><literal>live_router_check_interval</literal> - Specifies a time interval in seconds after
+ which the router checker will ping the live routers. The default value is 60. To set the
+ value to 50, enter:</para>
+ <screen>options lnet live_router_check_interval=50</screen>
+ </listitem>
+ <listitem>
+ <para><literal>dead_router_check_interval</literal> - Specifies a time interval in seconds
+ after which the router checker will check the dead routers. The default value is 60. To
+ set the value to 50, enter:</para>
+ <screen>options lnet dead_router_check_interval=50</screen>
+ <note>
+ <para>A kernel process called <literal>router_checker</literal> is started when either of the two
+ options above are set (thus, the <literal>router_checker</literal> is enabled by
+ default). It otians a list of next-hops from the routes entered and checks whether
+ next-hops are alive. In a multi-hop router configuration, these parameters can be set
+ on the routers as well, or on other node types. If no routes are entered, no checking
+ is done.</para>
+ </note>
+ </listitem>
+ <listitem>
+ <para><literal>router_ping_timeout</literal> - Specifies a timeout for the router checker
+ when it checks live or dead routers. The router checker sends a ping message to each
+ dead or live router once every <literal>dead_router_check_interval</literal> or
+ <literal>live_router_check_interval</literal> respectively. The default value is 50.
+ To set the value to 60, enter:</para>
+ <screen>options lnet router_ping_timeout=60</screen>
+ <note>
<para>The <literal>router_ping_timeout</literal> is consistent with the default LND timeouts. You may have to increase it on very large clusters if the LND timeout is also increased. For larger clusters, we suggest increasing the check interval.</para>
- </note>
- </listitem>
- <listitem>
- <para><literal>check_routers_before_use</literal> - Specifies that routers are to be checked before use. Set to off by default. If this parameter is set to on, the dead_router_check_interval parameter must be given a positive integer value.</para>
- <screen>options lnet check_routers_before_use=on</screen>
- </listitem>
- </itemizedlist>
- <para>The router checker obtains the following information from each router:</para>
+ </note>
+ </listitem>
+ <listitem>
+ <para><literal>check_routers_before_use</literal> - Specifies that routers are to be checked
+ before use. Set to off by default. If this parameter is set to on, the
+ <literal>dead_router_check_interval</literal> parameter must be given a positive
+ integer value.</para>
+ <screen>options lnet check_routers_before_use=on</screen>
+ </listitem>
+ </itemizedlist>
+ <para>The <literal>router_checker</literal> obtains the following information from each router:</para>
<itemizedlist>
- <listitem>
+ <listitem>
<para> Time the router was disabled</para>
- </listitem>
- <listitem>
+ </listitem>
+ <listitem>
<para> Elapsed disable time</para>
- </listitem>
+ </listitem>
</itemizedlist>
- <para>If the router checker does not get a reply message from the router within router_ping_timeout seconds, it considers the router to be down.</para>
- <para>If a router is marked 'up' and responds to a ping, the timeout is reset.</para>
- <para>If 100 packets have been sent successfully through a router, the sent-packets counter for that router will have a value of 100.</para>
+ <para>If the <literal>router_checker</literal> does not get a reply message from the router within <literal>router_ping_timeout</literal> seconds, it considers the router to be down.</para>
+ <para>When a router in a priority class goes down, the traffic stops intermittently until LNET
+ safely marks the router which is down as 'down' and then proceeds on again, depending either
+ on other routers of the same or a different priority class. The time it takes for LNET to
+ recover is roughly based on the values for the <literal>live/dead_router_checker</literal>
+ parameters provided.</para>
+ <para>If a router that is marked 'up' responds to a ping, the timeout is reset. If 100 packets
+ have been sent successfully through a router, the sent-packets counter for that router will
+ have a value of 100. The ping response also provides the status of the NIDs of the node
+ being pinged. In this way, the pinging node knows whether to keep using this node as a
+ next-hop or not. If one of the NIDs of the router is down and the
+ <literal>avoid_asym_router_failure = 1</literal> is set, then that router is no longer
+ used.</para>
</section>
<section xml:id="dbdoclet.50438216_15200">
- <title><indexterm><primary>LNET</primary><secondary>best practice</secondary></indexterm>Best Practices for LNET Options</title>
+ <title><indexterm>
+ <primary>LNet</primary>
+ <secondary>best practice</secondary>
+ </indexterm>Best Practices for LNET Options</title>
<para>For the <literal>networks</literal>, <literal>ip2nets</literal>, and <literal>routes</literal> options, follow these best practices to avoid configuration errors.</para>
<section remap="h5">
- <title><indexterm><primary>LNET</primary><secondary>escaping commas with quotes</secondary></indexterm>Escaping commas with quotes</title>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>escaping commas with quotes</secondary>
+ </indexterm>Escaping commas with quotes</title>
+ <para>It is recommended that you use network addresses rather than host names to make it easier to read debug logs and debug configurations with multiple interfaces.</para>
+ <para>If the server has multiple interfaces on the same subnet, the Linux kernel will send all traffic using the first configured interface. This is a limitation of Linux, not Lustre. In this case, network interface bonding should be used. For more information about network interface bonding, see <xref linkend="settingupbonding"/>.</para>
<para>Depending on the Linux distribution, commas may need to be escaped using single or double quotes. In the extreme case, the <literal>options</literal> entry would look like this:</para>
<para><screen>options lnet'networks="tcp0,elan0"' 'routes="tcp [2,10]@elan0"'</screen></para>
<para>Added quotes may confuse some distributions. Messages such as the following may indicate an issue related to added quotes:</para>
<para><screen>lnet: Unknown parameter 'networks'</screen></para>
- <para>A <literal>'Refusing connection - no matching NID'</literal> message generally points to an error in the LNET module configuration.</para>
+ <para>A <literal>'Refusing connection - no matching NID'</literal> message generally
+ points to an error in the LNET module configuration.</para>
</section>
<section remap="h5">
- <title><indexterm><primary>LNET</primary><secondary>comments</secondary></indexterm>Including comments</title>
- <para><emphasis>Place the semicolon terminating a comment immediately after the comment.</emphasis> LNET silently ignores everything between the <literal>#</literal> character at the beginning of the comment and the next semicolon.</para>
- <para>In this <emphasis>incorrect</emphasis> example, LNET silently ignores <literal>pt11 192.168.0.[92,96]</literal>, resulting in these nodes not being properly initialized. No error message is generated.</para>
- <screen>options lnet ip2nets="pt10 192.168.0.[89,93]; # comment with semicolon BEFORE comment \
-pt11 192.168.0.[92,96];</screen>
+ <title><indexterm><primary>LNet</primary><secondary>comments</secondary></indexterm>Including comments</title>
+ <para><emphasis>Place the semicolon terminating a comment immediately after the
+ comment.</emphasis> LNET silently ignores everything between the <literal>#</literal>
+ character at the beginning of the comment and the next semicolon.</para>
+ <para>In this <emphasis>incorrect</emphasis> example, LNET silently ignores <literal>pt11
+ 192.168.0.[92,96]</literal>, resulting in these nodes not being properly initialized. No
+ error message is generated.</para>
+ <screen>options lnet ip2nets="pt10 192.168.0.[89,93]; # comment with semicolon \
+ BEFORE comment pt11 192.168.0.[92,96];</screen>
<para>This <emphasis role="italic">correct</emphasis> example shows the required syntax:
</para>
<para><screen>options lnet ip2nets="pt10 192.168.0.[89,93] \
<indexterm><primary>tuning</primary></indexterm>
<indexterm><primary>tuning</primary><secondary>service threads</secondary></indexterm>
Optimizing the Number of Service Threads</title>
- <para>An OSS can have a minimum of 2 service threads and a maximum of 512 service threads. The number of service threads is a function of how much RAM and how many CPUs are on each OSS node (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will be started in order to process more requests concurrently, up to 4x the initial number of threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is 32 and the maximum thread count is 128.</para>
+ <para>An OSS can have a minimum of two service threads and a maximum of 512 service threads. The
+ number of service threads is a function of how much RAM and how many CPUs are on each OSS node
+ (1 thread / 128MB * num_cpus). If the load on the OSS node is high, new service threads will
+ be started in order to process more requests concurrently, up to 4x the initial number of
+ threads (subject to the maximum of 512). For a 2GB 2-CPU system, the default thread count is
+ 32 and the maximum thread count is 128.</para>
<para>Increasing the size of the thread pool may help when:</para>
<itemizedlist>
<listitem>
</section>
<section xml:id="dbdoclet.50438272_73839">
<title>
- <indexterm><primary>LNET</primary><secondary>tuning</secondary>
- </indexterm><indexterm><primary>tuning</primary><secondary>LNET</secondary></indexterm>Tuning LNET Parameters</title>
- <para>This section describes LNET tunables. that may be necessary on some systems to improve performance. To test the performance of your Lustre network, see <xref linkend='lnetselftest'/>.</para>
+ <indexterm>
+ <primary>LNet</primary>
+ <secondary>tuning</secondary>
+ </indexterm><indexterm>
+ <primary>tuning</primary>
+ <secondary>LNet</secondary>
+ </indexterm>Tuning LNET Parameters</title>
+ <para>This section describes LNET tunables, the use of which may be necessary on some systems to
+ improve performance. To test the performance of your Lustre network, see <xref linkend='lnetselftest'/>.</para>
<section remap="h3">
<title>Transmit and Receive Buffer Size</title>
<para>The kernel allocates buffers for sending and receiving messages on a network.</para>
<para>By default, this parameter is off. As always, you should test the performance to compare the impact of changing this parameter.</para>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>Network interface binding</secondary></indexterm>Binding Network Interface Against CPU Partitions</title>
- <para condition="l23">Lustre 2.3 and beyond provide enhanced network interface control. The
- enhancement means that an administrator can bind an interface to one or more CPU Partitions.
- Bindings are specified as options to the lnet modules. For more information on specifying
- module options, see <xref linkend="dbdoclet.50438293_15350"/></para>
-<para>For example, <literal>o2ib0(ib0)[0,1]</literal> will ensure that all messages for <literal>o2ib0</literal> will be handled by LND threads executing on <literal>CPT0</literal> and <literal>CPT1</literal>. An additional example might be: <literal>tcp1(eth0)[0]</literal>. Messages for <literal>tcp1</literal> are handled by threads on <literal>CPT0</literal>.</para>
+ <para condition="l23">Lustre 2.3 and beyond provide enhanced network interface control. The enhancement means that
+ an administrator can bind an interface to one or more CPU partitions. Bindings are specified
+ as options to the LNET modules. For more information on specifying module options, see <xref
+ linkend="dbdoclet.50438293_15350"/></para>
+<para condition="l23">For example, <literal>o2ib0(ib0)[0,1]</literal> will ensure that all messages
+ for <literal>o2ib0</literal> will be handled by LND threads executing on
+ <literal>CPT0</literal> and <literal>CPT1</literal>. An additional example might be:
+ <literal>tcp1(eth0)[0]</literal>. Messages for <literal>tcp1</literal> are handled by
+ threads on <literal>CPT0</literal>.</para>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>Network interface credits</secondary></indexterm>Network Interface Credits</title>
- <para>Network interface (NI) credits are shared across all CPU partitions (CPT). For example, a machine has 4 CPTs and NI credits is 512, then each partition will has 128 credits. If a large number of CPTs exist on the system, LNet will check and validate the NI credits value for each CPT to ensure each CPT has workable number of credits. For example, a machine has 16 CPTs and NI credits is set to 256, then each partition only has 16 credits. 16 NI credits is low and could negatively impact performance. As a result, LNet will automatically make an adjustment to 8*peer_credits (peer_credits is 8 by default), so credits for each partition is still 64.</para>
- <para>Modifying the NI Credit count can be performed by an administrator using <literal>ksoclnd</literal> or <literal>ko2iblnd</literal>. For example:</para>
+ <para>Network interface (NI) credits are shared across all CPU partitions (CPT). For example, a
+ machine has 4 CPTs and NI credits is 512, then each partition will has 128 credits. If a
+ large number of CPTs exist on the system, LNET will check and validate the NI credits value
+ for each CPT to ensure each CPT has workable number of credits. For example, a machine has
+ 16 CPTs and NI credits is set to 256, then each partition only has 16 credits. 16 NI credits
+ is low and could negatively impact performance. As a result, LNET will automatically make an
+ adjustment to 8*peer_credits (peer_credits is 8 by default), so credits for each partition
+ is still 64.</para>
+ <para>Modifying the NI credit count can be performed by an administrator using
+ <literal>ksoclnd</literal> or <literal>ko2iblnd</literal>. For example:</para>
<screen>ksocklnd credits=256</screen>
- <para>applies 256 credits to TCP connections. Applying 256 credits to IB connections can be achieved with:</para>
+ <para>In this example, 256 credits are applied to TCP connections. Applying 256 credits to IB
+ connections can be achieved with:</para>
<screen>ko2iblnd credits=256</screen>
- <note condition='l23'><para>In Lustre 2.3 and beyond, LNET may revalidate the NI Credits so that the administrator's
- request does not persist.</para></note>
+ <note><para condition="l23">From Lustre 2.3 and beyond, it is possible that LNET may
+ revalidate the NI Credits so that the administrator's request does not persist.</para></note>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>router buffers</secondary></indexterm>Router Buffers</title>
- <para>When a node is set up as an LNET router, three pools of buffers are allocated: tiny,
- small and large. These pools are allocated per CPU partition and are used to buffer messages
- that arrive at the router to be forwarded to the next hop. The three different buffer sizes
- accommodate different size messages. If a message arrives that can fit in a tiny buffer then
- a tiny buffer is used, if a message doesn’t fit in a tiny buffer, but fits in a small
- buffer, then a small buffer is used. Finally if a message doesn’t fit in either a tiny
- buffer or a small buffer, a large buffer is used.</para>
- <para>Router buffers are shared by all CPU partitions. For a machine with a large number of CPTs,
- the router buffer number may need to be specified manually for best performance. A low
- number of router buffers risks starving the CPU partitions of resources.</para>
- <para>The default setting for router buffers typically results in acceptable performance. LNET
- automatically sets a default value to reduce the likelihood of resource starvation. The size
- of a router buffer can be modified as shown in the example below. In this example, the size
- of the large buffer is modified using the <literal>large_router_buffers</literal>
- parameter.</para>
- <screen>lnet large_router_buffers=8192</screen>
- <note condition='l23'><para>In Lustre 2.3 and beyond, LNET may revalidate the NI Credits so that the administrator's
- request does not persist.</para></note>
+ <para>Router buffers are shared by all CPU partitions and the client nodes accessing the router.
+ For a machine with a large number of CPTs, the router buffer number may need to be specified
+ manually for best performance. A low number of router buffers risks starving the CPU
+ partitions of resources.</para>
+ <para>The default setting for router buffers will typically perform well. LNET automatically sets a
+ default value to reduce the likelihood of resource starvation</para>
+ <para>An administrator may modify router buffers using the required parameter.</para>
+ <para>There are three different kinds of router buffer parameters:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>tiny_router_buffers</literal>: Zero payload buffers which are used for signals and acknowledgements.</para>
+ </listitem>
+ <listitem>
+ <para><literal>small_router_buffers</literal>: 4KB payload buffers for small messages</para>
+ </listitem>
+ <listitem>
+ <para><literal>large_router_buffers</literal>: 1MB maximum payload buffers. This corresponds to the recommended size of 1MB for an RPC.</para>
+ </listitem>
+ </itemizedlist>
+ <para>These can be set using this syntax:</para>
+ <screen>options lnet large_router_buffers=8192</screen>
+ <note><para condition=""l23"">From Lustre 2.3 and beyond, it is possible that LNET may
+ revalidate the router buffer setting and the administrator's request do not
+ persist.</para></note>
+ <para>The purpose of the router buffers is to create a system of quotas where no single client can overwhelm the router. Router buffers are allotted to the clients.</para>
+ <para>So if increasing the router buffer parameters increases the total number of router buffers
+ available, the parameter <literal>peer_buffer_credit</literal> denotes the number of buffers
+ allotted to each client individually.</para>
+ <para>It can be set in a similar manner.</para>
+ <screen>options lnet peer_buffer_credits=256</screen>
</section>
<section><title><indexterm><primary>tuning</primary><secondary>portal round-robin</secondary></indexterm>Portal Round-Robin</title>
- <para>Portal round-robin defines the policy LNet applies to deliver events and messages to the upper layers. The upper layers are ptlrpc service or LNet selftest.</para>
- <para>If portal round-robin is disabled, LNet will deliver messages to CPTs based on a hash of the source NID. Hence, all messages from a specific peer will be handled by the same CPT. This can reduce data traffic between CPUs. However, for some workloads, this behavior may result in poorly balancing loads across the CPU.</para>
- <para>If portal round-robin is enabled, LNet will round-robin incoming events across all CPTs. This may balance load better across the CPU but can incur a cross CPU overhead.</para>
+ <para>Portal round-robin defines the policy LNET applies to deliver events and messages to the
+ upper layers. The upper layers are PLRPC service or LNET selftest.</para>
+ <para>If portal round-robin is disabled, LNET will deliver messages to CPTs based on a hash of the
+ source NID. Hence, all messages from a specific peer will be handled by the same CPT. This
+ can reduce data traffic between CPUs. However, for some workloads, this behavior may result
+ in poorly balancing loads across the CPU.</para>
+ <para>If portal round-robin is enabled, LNET will round-robin incoming events across all CPTs. This
+ may balance load better across the CPU but can incur a cross CPU overhead.</para>
<para>The current policy can be changed by an administrator with <literal>echo <replaceable>value</replaceable> > /proc/sys/lnet/portal_rotor</literal>. There are four options for <literal><replaceable>value</replaceable></literal>:</para>
<itemizedlist>
<listitem>
</screen>
<para>For PTLRPC services that support high-priority RPCs, you can also supply an optional <replaceable>reg|hp</replaceable> token, in order to enable an NRS policy for handling only regular or high-priority RPCs on a given PTLRPC service, by running:</para>
<screen>lctl set_param {service}.nrs_policies="<replaceable>policy_name</replaceable> <replaceable>reg|hp</replaceable>"</screen>
- <para>For example, to enable the TRR policy for handling only regular, but not high-priority RPCs on the ost_io service, run:</para>
+ <para>For example, to enable the TRR policy for handling only regular, but not high-priority
+ RPCs on the <literal>ost_io</literal> service, run:</para>
<screen>$ lctl set_param ost.OSS.ost_io.nrs_policies="trr reg"
ost.OSS.ost_io.nrs_policies="trr reg"
</screen>
git://git.whamcloud.com / doc / manual.git / commitdiff