-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='managinglnet'>
- <info>
- <title xml:id='managinglnet.title'>Managing Lustre Networking (LNET)</title>
- </info>
-
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managinglnet">
+ <title xml:id="managinglnet.title">Managing Lustre Networking (LNET)</title>
<para>This chapter describes some tools for managing Lustre Networking (LNET) and includes the following sections:</para>
- <itemizedlist><listitem>
- <para><xref linkend="dbdoclet.50438203_51732"/>Updating the Health Status of a Peer or Router</para>
+ <itemizedlist>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438203_51732"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438203_48703"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438203_82542"/></para>
+ </listitem>
+ <listitem>
+ <para><xref linkend="dbdoclet.50438203_78227"/></para>
+ </listitem>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438203_51732">
+ <title><indexterm><primary>LNET</primary><secondary>management</secondary></indexterm>
+ Updating the Health Status of a Peer or Router</title>
+ <para>There are two mechanisms to update the health status of a peer or a router:</para>
+ <itemizedlist>
+ <listitem>
+ <para>LNET can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set <literal>auto_down</literal> and if desired <literal>check_routers_before_use</literal>. This initial check may cause a pause equal to <literal>router_ping_timeout</literal> at system startup, if there are dead routers in the system.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438203_48703"/>Starting and Stopping LNET</para>
+ <para>When there is a communication error, all LNDs notify LNET that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNET module parameter <literal>auto_down</literal> to <literal>0</literal>, LNET ignores all such peer-down notifications.</para>
</listitem>
+ </itemizedlist>
+ <para>Several key differences in both mechanisms:</para>
+ <itemizedlist>
<listitem>
- <para><xref linkend="dbdoclet.50438203_82542"/>Multi-Rail Configurations with LNET</para>
+ <para>The router pinger only checks routers for their health, while LNDs notices all dead peers, regardless of whether they are a router or not.</para>
</listitem>
<listitem>
- <para><xref linkend="dbdoclet.50438203_78227"/>Load Balancing with InfiniBand</para>
+ <para>The router pinger actively checks the router health by sending pings, but LNDs only notice a dead peer when there is network traffic going on.</para>
</listitem>
- </itemizedlist>
-
- <section xml:id="dbdoclet.50438203_51732">
- <title>15.1 Updating the Health Status of a Peer or <anchor xml:id="dbdoclet.50438203_marker-1288828" xreflabel=""/>Router</title>
- <para>There are two mechanisms to update the health status of a peer or a router:</para>
- <itemizedlist><listitem>
- <para>LNET can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set auto_down and if desired check_routers_before_use. This initial check may cause a pause equal to router_ping_timeout at system startup, if there are dead routers in the system.</para>
- </listitem>
-<listitem>
- <para>When there is a communication error, all LNDs notify LNET that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNET module parameter auto_down to 0, LNET ignores all such peer-down notifications.</para>
- </listitem>
-</itemizedlist>
- <para>Several key differences in both mechanisms:</para>
- <itemizedlist><listitem>
- <para>The router pinger only checks routers for their health, while LNDs notices all dead peers, regardless of whether they are a router or not.</para>
- </listitem>
-<listitem>
- <para>The router pinger actively checks the router health by sending pings, but LNDs only notice a dead peer when there is network traffic going on.</para>
- </listitem>
-<listitem>
- <para> The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.</para>
- </listitem>
-</itemizedlist>
- </section>
- <section xml:id="dbdoclet.50438203_48703">
- <title>15.2 Starting and Stopping LNET</title>
- <para>Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre.</para>
- <section remap="h3">
- <title>15.2.1 Starting <anchor xml:id="dbdoclet.50438203_marker-1287400" xreflabel=""/>LNET</title>
- <para>To start LNET, run:</para>
- <screen>$ modprobe lnet
-$ lctl network up
-</screen>
- <para>To see the list of local NIDs, run:</para>
- <screen>$ lctl list_nids
-</screen>
- <para>This command tells you the network(s) configured to work with Lustre</para>
- <para>If the networks are not correctly setup, see the modules.conf "networks=" line and make sure the network layer modules are correctly installed and configured.</para>
- <para>To get the best remote NID, run:</para>
- <screen>$ lctl which_nid <NID list>
-</screen>
- <para>where <NID list> is the list of available NIDs.</para>
- <para>This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node.</para>
- <section remap="h4">
- <title>15.2.1.1 <anchor xml:id="dbdoclet.50438203_46145" xreflabel=""/>Starting Clients</title>
- <para>To start a TCP client, run:</para>
- <screen>mount -t lustre mdsnode:/mdsA/client /mnt/lustre/
-</screen>
- <para>To start an Elan client, run:</para>
- <screen>mount -t lustre 2@elan0:/mdsA/client /mnt/lustre
-</screen>
- </section>
- </section>
- <section remap="h3">
- <title>15.2.2 Stopping <anchor xml:id="dbdoclet.50438203_marker-1288543" xreflabel=""/>LNET</title>
- <para>Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run:</para>
- <screen>lctl network unconfigure
-</screen>
-
-<note>
- <para>
-Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
-</note>
-
- <para>To unconfigure the LNET network, run:</para>
- <screen>modprobe -r <any lnd and the lnet modules>
-</screen>
-
-<note>
- <para>
-To remove all Lustre modules, run:</para><para>$ lctl modules | awk '{print $2}' | xargs rmmod</para>
-</note>
-
-
+ <listitem>
+ <para>The router pinger can bring a router from alive to dead or vice versa, but LNDs can only bring a peer down.</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="dbdoclet.50438203_48703">
+ <title><indexterm><primary>LNET</primary><secondary>starting/stopping</secondary></indexterm>Starting and Stopping LNET</title>
+ <para>Lustre automatically starts and stops LNET, but it can also be manually started in a standalone manner. This is particularly useful to verify that your networking setup is working correctly before you attempt to start Lustre.</para>
+ <section remap="h3">
+ <title>Starting LNET</title>
+ <para>To start LNET, run:</para>
+ <screen>$ modprobe lnet
+$ lctl network up</screen>
+ <para>To see the list of local NIDs, run:</para>
+ <screen>$ lctl list_nids</screen>
+ <para>This command tells you the network(s) configured to work with Lustre</para>
+ <para>If the networks are not correctly setup, see the <literal>modules.conf</literal> "<literal>networks=</literal>" line and make sure the network layer modules are correctly installed and configured.</para>
+ <para>To get the best remote NID, run:</para>
+ <screen>$ lctl which_nid <NID list></screen>
+ <para>where <literal><NID list></literal> is the list of available NIDs.</para>
+ <para>This command takes the "best" NID from a list of the NIDs of a remote host. The "best" NID is the one that the local node uses when trying to communicate with the remote node.</para>
+ <section remap="h4">
+ <title>Starting Clients</title>
+ <para>To start a TCP client, run:</para>
+ <screen>mount -t lustre mdsnode:/mdsA/client /mnt/lustre/</screen>
+ <para>To start an Elan client, run:</para>
+ <screen>mount -t lustre 2@elan0:/mdsA/client /mnt/lustre</screen>
</section>
</section>
- <section xml:id="dbdoclet.50438203_72197">
- <title>15.3 <anchor xml:id="dbdoclet.50438203_82542" xreflabel=""/>Multi-Rail Configurations with <anchor xml:id="dbdoclet.50438203_Multi-rail-configurations-with-LNET-LNET" xreflabel=""/>LNET</title>
- <para>To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) <footnote><para>Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.</para></footnote> using LNET, consider these points:</para>
- <itemizedlist><listitem>
- <para> LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
- </listitem>
-<listitem>
- <para> Multi-rail LNET configurations do not provide an additional level of network fault tolerance. The configurations described below are for bandwidth aggregation only. Network interface failover is planned as an upcoming Lustre feature.</para>
- </listitem>
-<listitem>
- <para> A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are:</para>
- <itemizedlist><listitem>
- <para> Fewest hops (to minimize routing), and</para>
- </listitem>
-<listitem>
- <para> Appears first in the "networks" or "ip2nets" LNET configuration strings</para>
- </listitem>
-</itemizedlist>
- </listitem>
-</itemizedlist>
+ <section remap="h3">
+ <title>Stopping LNET</title>
+ <para>Before the LNET modules can be removed, LNET references must be removed. In general, these references are removed automatically when Lustre is shut down, but for standalone routers, an explicit step is needed to stop LNET. Run:</para>
+ <screen>lctl network unconfigure</screen>
+ <note>
+ <para>Attempting to remove Lustre modules prior to stopping the network may result in a crash or an LNET hang. if this occurs, the node must be rebooted (in most cases). Make sure that the Lustre network and Lustre are stopped prior to unloading the modules. Be extremely careful using rmmod -f.</para>
+ </note>
+ <para>To unconfigure the LNET network, run:</para>
+ <screen>modprobe -r <any lnd and the lnet modules></screen>
+ <note>
+ <para>
+To remove all Lustre modules, run:</para>
+ <para><literal>$ lctl modules | awk '{print $2}' | xargs rmmod</literal></para>
+ </note>
</section>
- <section xml:id="dbdoclet.50438203_78227">
- <title>15.4 Load Balancing with InfiniBand</title>
- <para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.</para>
- <section remap="h3">
- <title>15.4.1 Setting Up modprobe.conf<anchor xml:id="dbdoclet.50438203_77834" xreflabel=""/><anchor xml:id="dbdoclet.50438203_marker-1290316" xreflabel=""/> for Load Balancing</title>
- <para>To configure LNET for load balancing on clients and servers:</para>
- <para> 1. Set the modprobe.conf options.</para>
- <para>Depending on your configuration, set modprobe.conf options as follows:</para>
- <itemizedlist><listitem>
- <para> Dual HCA OSS server</para>
- </listitem>
-</itemizedlist>
- <screen>options lnet networks="o2ib0(ib0),o2ib1(ib1) 192.168.10.1.[101-102]
-</screen>
- <itemizedlist><listitem>
- <para> Client with the odd IP address</para>
+ </section>
+ <section xml:id="dbdoclet.50438203_82542">
+ <title><indexterm><primary>LNET</primary><secondary>multi-rail configuration</secondary></indexterm>Multi-Rail Configurations with LNET</title>
+ <para>To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) <footnote>
+ <para>Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.</para>
+ </footnote> using LNET, consider these points:</para>
+ <itemizedlist>
+ <listitem>
+ <para>LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
+ </listitem>
+ <listitem>
+ <para>Multi-rail LNET configurations do not provide an additional level of network fault tolerance. The configurations described below are for bandwidth aggregation only. Network interface failover is planned as an upcoming Lustre feature.</para>
+ </listitem>
+ <listitem>
+ <para>A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Fewest hops (to minimize routing), and</para>
</listitem>
-</itemizedlist>
- <screen>options lnet networks=o2ib0(ib0) 192.168.10.[103-253/2]
-</screen>
- <itemizedlist><listitem>
- <para> Client with the even IP address</para>
+ <listitem>
+ <para>Appears first in the "<literal>networks</literal>" or "<literal>ip2nets</literal>" LNET configuration strings</para>
</listitem>
-</itemizedlist>
- <screen>options lnet networks=o2ib1(ib0) 192.168.10.[102-254/2]
+ </itemizedlist>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section xml:id="dbdoclet.50438203_78227">
+ <title><indexterm><primary>LNET</primary><secondary>Infiniband load balancing</secondary></indexterm>Load Balancing with InfiniBand</title>
+ <para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.</para>
+ <section remap="h3">
+ <title><indexterm><primary>LNET</primary><secondary>lustre.conf</secondary></indexterm>Setting Up <literal>lustre.conf</literal> for Load Balancing</title>
+ <para>To configure LNET for load balancing on clients and servers:</para>
+ <orderedlist>
+ <listitem>
+ <para>Set the <literal>lustre.conf</literal> options.</para>
+ <para>Depending on your configuration, set <literal>lustre.conf</literal> options as follows:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Dual HCA OSS server</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet networks="o2ib0(ib0),o2ib1(ib1) 192.168.10.1.[101-102] </screen>
+ <itemizedlist>
+ <listitem>
+ <para>Client with the odd IP address</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet networks=o2ib0(ib0) 192.168.10.[103-253/2] </screen>
+ <itemizedlist>
+ <listitem>
+ <para>Client with the even IP address</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet networks=o2ib1(ib0) 192.168.10.[102-254/2]
</screen>
- <para> 2. Run the modprobe lnet command and create a combined MGS/MDT file system.</para>
- <para>The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS).</para>
- <screen>modprobe lnet
+ </listitem>
+ <listitem>
+ <para>Run the modprobe lnet command and create a combined MGS/MDT file system.</para>
+ <para>The following commands create the MGS/MDT file system and mount the servers (MGS/MDT and OSS).</para>
+ <screen>modprobe lnet
$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
$ mkdir -p <mount point>
$ mount -t lustre <block device> <mount point>
$ mkfs.lustre --fsname lustre --mgs --mdt <block device name>
$ mkdir -p <mount point>
$ mount -t lustre <block device> <mount point>
-$ mount -t lustre <block device> <mount point>
-</screen>
- <para>For example:</para>
- <screen>modprobe lnet
+$ mount -t lustre <block device> <mount point> </screen>
+ <para>For example:</para>
+ <screen>modprobe lnet
$ mkfs.lustre --fsname lustre --mdt --mgs /dev/sda
$ mkdir -p /mnt/test/mdt
$ mount -t lustre /dev/sda /mnt/test/mdt
$ mount -t lustre mgs@o2ib0:/lustre /mnt/mdt
-$ mkfs.lustre --fsname lustre --ost --mgsnode=mds@o2ib0 /dev/sda
+$ mkfs.lustre --fsname lustre --mgsnode=mds@o2ib0 --ost --index=0 /dev/sda
$ mkdir -p /mnt/test/mdt
$ mount -t lustre /dev/sda /mnt/test/ost
-$ mount -t lustre mgs@o2ib0:/lustre /mnt/ost
-</screen>
- <para> 3. Mount the clients.</para>
- <screen>mount -t lustre <MGS node>:/<fsname> <mount point>
-</screen>
- <para>This example shows an IB client being mounted.</para>
- <screen>mount -t lustre
-192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre
-</screen>
- <para>As an example, consider a two-rail IB cluster running the OFA stack (OFED) with these IPoIB address assignments.</para>
- <screen> ib0 ib1
+$ mount -t lustre mgs@o2ib0:/lustre /mnt/ost0</screen>
+ </listitem>
+ <listitem>
+ <para>Mount the clients.</para>
+ <screen>mount -t lustre <MGS node>:/<fsname> <mount point></screen>
+ <para>This example shows an IB client being mounted.</para>
+ <screen>mount -t lustre
+192.168.10.101@o2ib0,192.168.10.102@o2ib1:/mds/client /mnt/lustre</screen>
+ </listitem>
+ </orderedlist>
+ <para>As an example, consider a two-rail IB cluster running the OFA stack (OFED) with these IPoIB address assignments.</para>
+ <screen> ib0 ib1
Servers 192.168.0.* 192.168.1.*
-Clients 192.168.[2-127].* 192.168.[128-253].*
-</screen>
- <para>You could create these configurations:</para>
- <itemizedlist><listitem>
- <para> A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are the actual bottleneck.</para>
- </listitem>
-</itemizedlist>
- <screen>ip2nets="o2ib0(ib0), o2ib1(ib1) 192.168.[0-1].* \
+Clients 192.168.[2-127].* 192.168.[128-253].*</screen>
+ <para>You could create these configurations:</para>
+ <itemizedlist>
+ <listitem>
+ <para>A cluster with more clients than servers. The fact that an individual client cannot get two rails of bandwidth is unimportant because the servers are the actual bottleneck.</para>
+ </listitem>
+ </itemizedlist>
+ <screen>ip2nets="o2ib0(ib0), o2ib1(ib1) 192.168.[0-1].* \
#all servers;\
o2ib0(ib0) 192.168.[2-253].[0-252/2] #even cl\
ients;\
o2ib1(ib1) 192.168.[2-253].[1-253/2] #odd cli\
-ents"
-</screen>
- <para>This configuration gives every server two NIDs, one on each network, and statically load-balances clients between the rails.</para>
- <itemizedlist><listitem>
- <para> A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail).</para>
- </listitem>
-</itemizedlist>
- <screen>ip2nets=" o2ib0(ib0) 192.168.[0-1].[0-252/2] \
+ents"</screen>
+ <para>This configuration gives every server two NIDs, one on each network, and statically load-balances clients between the rails.</para>
+ <itemizedlist>
+ <listitem>
+ <para>A single client that must get two rails of bandwidth, and it does not matter if the maximum aggregate bandwidth is only (# servers) * (1 rail).</para>
+ </listitem>
+ </itemizedlist>
+ <screen>ip2nets=" o2ib0(ib0) 192.168.[0-1].[0-252/2] \
#even servers;\
o2ib1(ib1) 192.168.[0-1].[1-253/2] \
#odd servers;\
o2ib0(ib0),o2ib1(ib1) 192.168.[2-253].* \
- #clients"
-</screen>
- <para>This configuration gives every server a single NID on one rail or the other. Clients have a NID on both rails.</para>
- <itemizedlist><listitem>
- <para> All clients and all servers must get two rails of bandwidth.</para>
- </listitem>
-</itemizedlist>
- <screen>ip2nets=†o2ib0(ib0),o2ib2(ib1) 192.168.[0-1].[0-252/2] \
+ #clients"</screen>
+ <para>This configuration gives every server a single NID on one rail or the other. Clients have a NID on both rails.</para>
+ <itemizedlist>
+ <listitem>
+ <para>All clients and all servers must get two rails of bandwidth.</para>
+ </listitem>
+ </itemizedlist>
+ <screen>ip2nets=†o2ib0(ib0),o2ib2(ib1) 192.168.[0-1].[0-252/2] \
#even servers;\
o2ib1(ib0),o2ib3(ib1) 192.168.[0-1].[1-253/2] \
#odd servers;\
o2ib0(ib0),o2ib3(ib1) 192.168.[2-253].[0-252/2) \
#even clients;\
o2ib1(ib0),o2ib2(ib1) 192.168.[2-253].[1-253/2) \
-#odd clients"
-</screen>
- <para>This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with o2ib0 on rail0, and "odd" servers with o2ib3 on rail1. Similarly, it connects "odd" clients to "odd" servers with o2ib1 on rail0, and "even" servers with o2ib2 on rail1.</para>
- </section>
+#odd clients"</screen>
+ <para>This configuration includes two additional proxy o2ib networks to work around Lustre's simplistic NID selection algorithm. It connects "even" clients to "even" servers with <literal>o2ib0</literal> on <literal>rail0</literal>, and "odd" servers with <literal>o2ib3</literal> on <literal>rail1</literal>. Similarly, it connects "odd" clients to "odd" servers with <literal>o2ib1</literal> on <literal>rail0</literal>, and "even" servers with <literal>o2ib2</literal> on <literal>rail1</literal>.</para>
+ </section>
</section>
</chapter>
git://git.whamcloud.com / doc / manual.git / blobdiff