<para>Password-free remote access to nodes in the system (provided by <literal>ssh</literal> or <literal>rsh</literal>).</para>
</listitem>
<listitem>
- <para>LNET self-test completed to test that Lustre networking has been properly installed
+ <para>LNet self-test completed to test that Lustre networking has been properly installed
and configured. See <xref linkend="lnetselftest"/>.</para>
</listitem>
<listitem>
<section xml:id="dbdoclet.50438293_15350">
<title>
<indexterm><primary>configuring</primary></indexterm>
- <indexterm><primary>LNET</primary><see>configuring</see></indexterm>
+ <indexterm><primary>LNet</primary><see>configuring</see></indexterm>
Introduction</title>
- <para>LNET network hardware and routing are now configured via module parameters. Parameters should be specified in the <literal>/etc/modprobe.d/lustre.conf</literal>file, for example:</para>
+ <para>LNet network hardware and routing are now configured via module parameters. Parameters should be specified in the <literal>/etc/modprobe.d/lustre.conf</literal>file, for example:</para>
<screen>options lnet networks=tcp0(eth2)</screen>
<para>The above option specifies that this node should use the TCP protocol on the eth2 network interface.</para>
- <para>Module parameters are read when the module is first loaded. Type-specific LND modules (for instance, <literal>ksocklnd</literal>) are loaded automatically by the LNET module when LNET starts (typically upon <literal>modprobe ptlrpc</literal>).</para>
- <para>LNET configuration parameters can be viewed under <literal>/sys/module/lnet/parameters/</literal>, and LND-specific parameters under the name of the corresponding LND, for example <literal>/sys/module/ksocklnd/parameters/</literal> for the socklnd (TCP) LND.</para>
- <para>For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. Unmarked parameters can only be set when LNET loads for the first time. Changes to parameters marked with <literal>Wc</literal> only have effect when connections are established (existing connections are not affected by these changes.)</para>
+ <para>Module parameters are read when the module is first loaded. Type-specific LND modules (for instance, <literal>ksocklnd</literal>) are loaded automatically by the LNet module when LNet starts (typically upon <literal>modprobe ptlrpc</literal>).</para>
+ <para>LNet configuration parameters can be viewed under <literal>/sys/module/lnet/parameters/</literal>, and LND-specific parameters under the name of the corresponding LND, for example <literal>/sys/module/ksocklnd/parameters/</literal> for the socklnd (TCP) LND.</para>
+ <para>For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. Unmarked parameters can only be set when LNet loads for the first time. Changes to parameters marked with <literal>Wc</literal> only have effect when connections are established (existing connections are not affected by these changes.)</para>
</section>
<section xml:id="dbdoclet.50438293_78010">
<title>
<para>A separate <literal>lustre.conf</literal> file makes distributing the configuration much easier.</para>
</listitem>
<listitem>
- <para>If you set <literal>config_on_load=1</literal>, LNET starts at
+ <para>If you set <literal>config_on_load=1</literal>, LNet starts at
<literal>modprobe</literal> time rather than waiting for the Lustre file system to
start. This ensures routers start working at module load time.</para>
</listitem>
# lctl> net down</screen>
<itemizedlist>
<listitem>
- <para>Remember the <literal>lctl ping {nid}</literal> command - it is a handy way to check your LNET configuration.</para>
+ <para>Remember the <literal>lctl ping {nid}</literal> command - it is a handy way to check your LNet configuration.</para>
</listitem>
</itemizedlist>
<section remap="h3">
- <title><indexterm><primary>configuring</primary><secondary>LNET options</secondary></indexterm>
-LNET Options</title>
- <para>This section describes LNET options.</para>
+ <title><indexterm><primary>configuring</primary><secondary>LNet options</secondary></indexterm>
+LNet Options</title>
+ <para>This section describes LNet options.</para>
<section remap="h4">
<title><indexterm><primary>configuring</primary><secondary>network topology</secondary></indexterm>
Network Topology</title>
Lustre file system uses any IP addresses aliased to the loopback (by default). When in
doubt, explicitly specify networks.</para>
</note>
- <para><literal>ip2nets</literal> ("") is a string that lists globally-available networks, each with a set of IP address ranges. LNET determines the locally-available networks from this list by matching the IP address ranges with the local IPs of a node. The purpose of this option is to be able to use the same <literal>modules.conf</literal> file across a variety of nodes on different networks. The string has the following syntax.</para>
+ <para><literal>ip2nets</literal> ("") is a string that lists globally-available networks, each with a set of IP address ranges. LNet determines the locally-available networks from this list by matching the IP address ranges with the local IPs of a node. The purpose of this option is to be able to use the same <literal>modules.conf</literal> file across a variety of nodes on different networks. The string has the following syntax.</para>
<screen><ip2nets> :== <net-match> [ <comment> ] { <net-sep> <net-match> }
<net-match> :== [ <w> ] <net-spec> <w> <ip-range> { <w> <ip-range> }
[ <w> ]
<title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>forwarding</tertiary></indexterm>
forwarding ("")</title>
<para>This is a string that can be set either to "<literal>enabled</literal>" or "<literal>disabled</literal>" for explicit control of whether this node should act as a router, forwarding communications between all local networks.</para>
- <para>A standalone router can be started by simply starting LNET ('<literal>modprobe ptlrpc</literal>') with appropriate network topology options.</para>
+ <para>A standalone router can be started by simply starting LNet ('<literal>modprobe ptlrpc</literal>') with appropriate network topology options.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<para> <literal>acceptor</literal></para>
</entry>
<entry>
- <para>The acceptor is a TCP/IP service that some LNDs use to establish communications. If a local network requires it and it has not been disabled, the acceptor listens on a single port for connection requests that it redirects to the appropriate local network. The acceptor is part of the LNET module and configured by the following options:</para>
+ <para>The acceptor is a TCP/IP service that some LNDs use to establish communications. If a local network requires it and it has not been disabled, the acceptor listens on a single port for connection requests that it redirects to the appropriate local network. The acceptor is part of the LNet module and configured by the following options:</para>
<itemizedlist>
<listitem>
<para><literal>secure</literal> - Accept connections only from reserved TCP ports (below 1023).</para>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
- <title xml:id="configuringlnet.title">Configuring Lustre Networking (LNET)</title>
- <para>This chapter describes how to configure Lustre Networking (LNET). It includes the following sections:</para>
+ <title xml:id="configuringlnet.title">Configuring Lustre Networking (LNet)</title>
+ <para>This chapter describes how to configure Lustre Networking (LNet). It includes the following sections:</para>
<itemizedlist>
<listitem>
<para><xref linkend="dbdoclet.50438216_15201"/>
</listitem>
</itemizedlist>
<note>
- <para>Configuring LNET is optional.</para>
- <para> LNET will use the first TCP/IP interface it discovers on a
+ <para>Configuring LNet is optional.</para>
+ <para> LNet will use the first TCP/IP interface it discovers on a
system (<literal>eth0</literal>) if it's loaded using the
<literal>lctl network up</literal>. If this network configuration is
- sufficient, you do not need to configure LNET. LNet configuration is
+ sufficient, you do not need to configure LNet. LNet configuration is
required if you are using Infiniband or multiple Ethernet
interfaces.</para>
<para condition='l27'>The <literal>lnetctl</literal> utility can be used
- to initialize LNET without bringing up any network interfaces. This
- gives flexibility to the user to add interfaces after LNET has been
+ to initialize LNet without bringing up any network interfaces. This
+ gives flexibility to the user to add interfaces after LNet has been
loaded.</para>
<para condition='l27'>DLC also introduces a C-API to enable
- configuring LNET programatically. See <xref
+ configuring LNet programatically. See <xref
linkend="lnetconfigurationapi"/></para>
</note>
<section xml:id="dbdoclet.50438216_15201" condition='l27'>
<title><indexterm>
- <primary>LNET</primary>
- <secondary>Configuring LNET</secondary>
- </indexterm>Configuring LNET via <literal>lnetctl</literal></title>
+ <primary>LNet</primary>
+ <secondary>Configuring LNet</secondary>
+ </indexterm>Configuring LNet via <literal>lnetctl</literal></title>
<para>The <literal>lnetctl</literal> utility can be used to initialize
- and configure the LNET kernel module after it has been loaded via
+ and configure the LNet kernel module after it has been loaded via
<literal>modprobe</literal>. In general the lnetctl format is as
follows: </para>
<screen>lnetctl cmd subcmd [options]</screen>
<para>
<itemizedlist>
<listitem>
- <para>Configuring/unconfiguring LNET</para>
+ <para>Configuring/unconfiguring LNet</para>
</listitem>
<listitem>
<para>Adding/removing/showing Networks</para>
</para>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
- </indexterm>Configuring LNET</title>
- <para>After LNET has been loaded via <literal>modprobe</literal>,
- <literal>lnetctl</literal> utility can be used to configure LNET
+ </indexterm>Configuring LNet</title>
+ <para>After LNet has been loaded via <literal>modprobe</literal>,
+ <literal>lnetctl</literal> utility can be used to configure LNet
without bringing up networks which are specified in the module
parameters. It can also be used to configure network interfaces
specified in the module prameters by providing the
<screen>lnetctl lnet configure [--all]
# --all: load NI configuration from module parameters</screen>
<para>The <literal>lnetctl</literal> utility can also be used to
- unconfigure LNET.</para>
+ unconfigure LNet.</para>
<screen>lnetctl lnet unconfigure</screen>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
</indexterm>Adding, Deleting and Showing networks</title>
- <para>Networks can be added and deleted after the LNET kernel module
+ <para>Networks can be added and deleted after the LNet kernel module
is loaded.</para>
<screen>lnetctl net add: add a network
--net: net name (ex tcp0)
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
</indexterm>Adding, Deleting and Showing routes</title>
- <para>A set of routes can be added to identify how LNET messages are
+ <para>A set of routes can be added to identify how LNet messages are
to be routed.</para>
<screen>lnetctl route add: add a route
- --net: net name (ex tcp0) LNET message is destined to.
+ --net: net name (ex tcp0) LNet message is destined to.
The can not be a local network.
--gateway: gateway node nid (ex 10.1.1.2@tcp) to route
- all LNET messaged destined for the identified
+ all LNet messaged destined for the identified
network
--hop: number of hops to final destination
(1 < hops < 255)
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
</indexterm>Enabling and Disabling Routing</title>
- <para>When an LNET node is configured as a router it will route LNet
+ <para>When an LNet node is configured as a router it will route LNet
messages not destined to itself. This feature can be enabled or
disabled as follows.</para>
<screen>lnetctl set routing [0 | 1]
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
</indexterm>Showing routing information</title>
<para>When routing is enabled on a node, the tiny, small and large
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
</indexterm>Configuring Routing Buffers</title>
<para> The routing buffers values configured specify the number of
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
</indexterm>Importing YAML Configuration File</title>
<para>Configuration can be described in YAML format and can be fed
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
</indexterm>Exporting Configuration in YAML format</title>
<para><literal>lnetctl</literal> utility provides the 'export'
- command to dump current LNET configuration in YAML format </para>
+ command to dump current LNet configuration in YAML format </para>
<screen>lnetctl export FILE.yaml
lnetctl export > FILE.yaml</screen>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cli</secondary>
- </indexterm>Showing LNET Traffic Statistics</title>
- <para><literal>lnetctl</literal> utility can dump the LNET traffic
+ </indexterm>Showing LNet Traffic Statistics</title>
+ <para><literal>lnetctl</literal> utility can dump the LNet traffic
statistiscs as follows</para>
<screen>lnetctl stats show</screen>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>yaml syntax</secondary>
</indexterm>YAML Syntax</title>
<para>The <literal>lnetctl</literal> utility can take in a YAML file
operation.</para>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>network yaml syntax</secondary>
</indexterm>Network Configuration</title>
<para/>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>buffer yaml syntax</secondary>
</indexterm>Enable Routing and Adjust Router Buffer Configuration</title>
<para/>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>statistics yaml syntax</secondary>
</indexterm>Show Statistics</title>
<para/>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>router yaml syntax</secondary>
</indexterm>Route Configuration</title>
<para/>
</section>
</section>
<section xml:id="dbdoclet.50438216_33148">
- <title><indexterm><primary>LNET</primary></indexterm>
+ <title><indexterm><primary>LNet</primary></indexterm>
- Overview of LNET Module Parameters</title>
- <para>LNET kernel module (lnet) parameters specify how LNet is to be
+ Overview of LNet Module Parameters</title>
+ <para>LNet kernel module (lnet) parameters specify how LNet is to be
configured to work with Lustre, including which NICs will be
configured to work with Lustre and the routing to be used with
Lustre.</para>
- <para>Parameters for LNET can be specified in the
+ <para>Parameters for LNet can be specified in the
<literal>/etc/modprobe.d/lustre.conf</literal> file. In some cases
the parameters may have been stored in
<literal>/etc/modprobe.conf</literal>, but this has been deprecated
</listitem>
<listitem>
<para><literal>ip2nets</literal> - Lists globally-available
- networks, each with a range of IP addresses. LNET then identifies
+ networks, each with a range of IP addresses. LNet then identifies
locally-available networks through address list-matching
lookup.</para>
</listitem>
Lustre nodes to detect router health status, avoid routers that appear
dead, and reuse those that restore service after failures. See <xref
linkend="dbdoclet.50438216_35668"/> for more details.</para>
- <para>For a complete reference to the LNET module parameters, see
- <emphasis><xref linkend="configurationfilesmoduleparameters"/>LNET
+ <para>For a complete reference to the LNet module parameters, see
+ <emphasis><xref linkend="configurationfilesmoduleparameters"/>LNet
Options</emphasis>.</para>
<note>
<para>We recommend that you use 'dotted-quad' notation for
logs and debug configurations with multiple interfaces.</para>
</note>
<section>
- <title><indexterm><primary>LNET</primary><secondary>using
+ <title><indexterm><primary>LNet</primary><secondary>using
NID</secondary></indexterm>Using a Lustre Network Identifier (NID)
to Identify a Node</title>
<para>A Lustre network identifier (NID) is used to uniquely identify
</section>
</section>
<section xml:id="dbdoclet.50438216_46279">
- <title><indexterm><primary>LNET</primary><secondary>module parameters</secondary></indexterm>Setting the LNet Module networks Parameter</title>
+ <title><indexterm><primary>LNet</primary><secondary>module parameters</secondary></indexterm>Setting the LNet Module networks Parameter</title>
<para>If a node has more than one network interface, you'll
typically want to dedicate a specific interface to Lustre. You can do
this by including an entry in the <literal>lustre.conf</literal> file
- on the node that sets the LNET module <literal>networks</literal>
+ on the node that sets the LNet module <literal>networks</literal>
parameter:</para>
<screen>options lnet networks=<replaceable>comma-separated list of
networks</replaceable></screen>
another interface, even if multiple interfaces are available on the
same node.</para>
<note>
- <para>LNET lines in <literal>lustre.conf</literal> are only used by
+ <para>LNet lines in <literal>lustre.conf</literal> are only used by
the local node to determine what to call its interfaces. They are
not used for routing decisions.</para>
</note>
(<literal>lo0</literal>) interface. Lustre does not ignore IP
addresses aliased to the loopback. If you alias IP addresses to
the loopback interface, you must specify all Lustre networks using
- the LNET networks parameter.</para>
+ the LNet networks parameter.</para>
</note>
<note>
<para>If the server has multiple interfaces on the same subnet,
</section>
</section>
<section xml:id="dbdoclet.50438216_31414">
- <title><indexterm><primary>LNET</primary><secondary>ip2nets</secondary></indexterm>Setting the LNet Module ip2nets Parameter</title>
+ <title><indexterm><primary>LNet</primary><secondary>ip2nets</secondary></indexterm>Setting the LNet Module ip2nets Parameter</title>
<para>The <literal>ip2nets</literal> option is typically used when a
single, universal <literal>lustre.conf</literal> file is run on all
servers and clients. Each node identifies the locally available
<para>Note that the IP address patterns listed in the
<literal>ip2nets</literal> option are <emphasis>only</emphasis> used
to identify the networks that an individual node should instantiate.
- They are <emphasis>not</emphasis> used by LNET for any other
+ They are <emphasis>not</emphasis> used by LNet for any other
communications purpose.</para>
<para>For the example below, the nodes in the network have these IP
addresses:</para>
<screen>options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \
tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'</screen>
<para>Each entry in <literal>ip2nets</literal> is referred to as a 'rule'.</para>
- <para>The order of LNET entries is important when configuring servers.
+ <para>The order of LNet entries is important when configuring servers.
If a server node can be reached using more than one network, the first
network specified in <literal>lustre.conf</literal> will be
used.</para>
<para>Because <literal>svr1</literal> and <literal>svr2</literal>
- match the first rule, LNET uses <literal>eth0</literal> for
+ match the first rule, LNet uses <literal>eth0</literal> for
<literal>tcp0</literal> on those machines. (Although
<literal>svr1</literal> and <literal>svr2</literal> also match the
second rule, the first matching rule for a particular network is
network.</para>
</section>
<section xml:id="dbdoclet.50438216_71227">
- <title><indexterm><primary>LNET</primary><secondary>routes</secondary></indexterm>Setting
+ <title><indexterm><primary>LNet</primary><secondary>routes</secondary></indexterm>Setting
the LNet Module routes Parameter</title>
- <para>The LNET module routes parameter is used to identify routers in
+ <para>The LNet module routes parameter is used to identify routers in
a Lustre configuration. These parameters are set in
<literal>modprobe.conf</literal> on each Lustre node. </para>
<para>Routes are typically set to connect to segregated subnetworks
or to cross connect two different types of networks such as tcp and
o2ib</para>
- <para>The LNET routes parameter specifies a colon-separated list of
+ <para>The LNet routes parameter specifies a colon-separated list of
router definitions. Each route is defined as a network number,
followed by a list of routers:</para>
<screen>routes=<replaceable>net_type router_NID(s)</replaceable></screen>
<screen>options lnet 'ip2nets="tcp0 192.168.0.*; \
o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \
o2ib0 192.16.8.0.[1-8]@tcp0"'</screen>
- <para>All LNET routers that bridge two networks are equivalent. They
+ <para>All LNet routers that bridge two networks are equivalent. They
are not configured as primary or secondary, and the load is balanced
across all available routers.</para>
- <para>The number of LNET routers is not limited. Enough routers should
+ <para>The number of LNet routers is not limited. Enough routers should
be used to handle the required file serving bandwidth plus a 25
percent margin for headroom.</para>
<section>
- <title><indexterm><primary>LNET</primary><secondary>routing
+ <title><indexterm><primary>LNet</primary><secondary>routing
example</secondary></indexterm>Routing Example</title>
<para>On the clients, place the following entry in the
<literal>lustre.conf</literal> file</para>
</section>
</section>
<section xml:id="dbdoclet.50438216_10523">
- <title><indexterm><primary>LNET</primary><secondary>testing</secondary></indexterm>Testing
+ <title><indexterm><primary>LNet</primary><secondary>testing</secondary></indexterm>Testing
the LNet Configuration</title>
<para>After configuring Lustre Networking, it is highly recommended
- that you test your LNET configuration using the LNet Self-Test
+ that you test your LNet configuration using the LNet Self-Test
provided with the Lustre software. For more information about using
- LNET Self-Test, see <xref linkend="lnetselftest"/>.</para>
+ LNet Self-Test, see <xref linkend="lnetselftest"/>.</para>
</section>
<section xml:id="dbdoclet.50438216_35668">
- <title><indexterm><primary>LNET</primary><secondary>route
+ <title><indexterm><primary>LNet</primary><secondary>route
checker</secondary></indexterm>Configuring the Router Checker</title>
<para>In a Lustre configuration in which different types of networks,
such as a TCP/IP network and an Infiniband network, are connected by
routed configuration to monitor the status of the routers. In a
multi-hop routing configuration, router checkers can be configured on
routers to monitor the health of their next-hop routers.</para>
- <para>A router checker is configured by setting lnet parameters in
+ <para>A router checker is configured by setting LNet parameters in
<literal>lustre.conf</literal> by including an entry in this
form:</para>
<screen>options lnet
sent-packets counter for that router will have a value of 100.</para>
</section>
<section xml:id="dbdoclet.50438216_15200">
- <title><indexterm><primary>LNET</primary><secondary>best
- practice</secondary></indexterm>Best Practices for LNET
+ <title><indexterm><primary>LNet</primary><secondary>best
+ practice</secondary></indexterm>Best Practices for LNet
Options</title>
<para>For the <literal>networks</literal>, <literal>ip2nets</literal>,
and <literal>routes</literal> options, follow these best practices to
avoid configuration errors.</para>
<section>
- <title><indexterm><primary>LNET</primary><secondary>escaping commas
+ <title><indexterm><primary>LNet</primary><secondary>escaping commas
with quotes</secondary></indexterm>Escaping commas with
quotes</title>
<para>Depending on the Linux distribution, commas may need to be
the following may indicate an issue related to added quotes:</para>
<para><screen>lnet: Unknown parameter 'networks'</screen></para>
<para>A <literal>'Refusing connection - no matching
- NID'</literal> message generally points to an error in the LNET
+ NID'</literal> message generally points to an error in the LNet
module configuration.</para>
</section>
<section>
- <title><indexterm><primary>LNET</primary><secondary>comments</secondary></indexterm>Including
+ <title><indexterm><primary>LNet</primary><secondary>comments</secondary></indexterm>Including
comments</title>
<para><emphasis>Place the semicolon terminating a comment
- immediately after the comment.</emphasis> LNET silently ignores
+ immediately after the comment.</emphasis> LNet silently ignores
everything between the <literal>#</literal> character at the
beginning of the comment and the next semicolon.</para>
- <para>In this <emphasis>incorrect</emphasis> example, LNET silently
+ <para>In this <emphasis>incorrect</emphasis> example, LNet silently
ignores <literal>pt11 192.168.0.[92,96]</literal>, resulting in
these nodes not being properly initialized. No error message is
generated.</para>
<listitem>
<para>
<emphasis>Set</emphasis> lnet
- <emphasis>module parameters to specify how Lustre Networking (LNET) is
- to be configured to work with a Lustre file system and test the LNET
- configuration.</emphasis>LNET will, by default, use the first TCP/IP
+ <emphasis>module parameters to specify how Lustre Networking (LNet) is
+ to be configured to work with a Lustre file system and test the LNet
+ configuration.</emphasis>LNet will, by default, use the first TCP/IP
interface it discovers on a system. If this network configuration is
- sufficient, you do not need to configure LNET. LNET configuration is
+ sufficient, you do not need to configure LNet. LNet configuration is
required if you are using InfiniBand or multiple Ethernet
interfaces.</para>
</listitem>
</itemizedlist>
- <para>For information about configuring LNET, see
- <xref linkend="configuringlnet" />. For information about testing LNET, see
+ <para>For information about configuring LNet, see
+ <xref linkend="configuringlnet" />. For information about testing LNet, see
<xref linkend="lnetselftest" />.</para>
<itemizedlist>
<glossentry xml:id="lnd">
<glossterm>LND</glossterm>
<glossdef>
- <para>Lustre network driver. A code module that enables LNET support
+ <para>Lustre network driver. A code module that enables LNet support
over particular transports, such as TCP and various kinds of InfiniBand
networks.</para>
</glossdef>
</glossentry>
<glossentry xml:id="lnet">
- <glossterm>LNET</glossterm>
+ <glossterm>LNet</glossterm>
<glossdef>
<para>Lustre networking. A message passing network protocol capable of
- running and routing through various physical layers. LNET forms the
+ running and routing through various physical layers. LNet forms the
underpinning of LNETrpc.</para>
</glossdef>
</glossentry>
<glossterm>MDC</glossterm>
<glossdef>
<para>Metadata client. A Lustre client component that sends metadata
- requests via RPC over LNET to the metadata target (MDT).</para>
+ requests via RPC over LNet to the metadata target (MDT).</para>
</glossdef>
</glossentry>
<glossentry xml:id="mdd">
<glossentry xml:id="nioapi">
<glossterm>NIO API</glossterm>
<glossdef>
- <para>A subset of the LNET RPC module that implements a library for
+ <para>A subset of the LNet RPC module that implements a library for
sending large network requests, moving buffers with RDMA.</para>
</glossdef>
</glossentry>
<glossentry xml:id="portal">
<glossterm>Portal</glossterm>
<glossdef>
- <para>A service address on an LNET NID that binds requests to a
+ <para>A service address on an LNet NID that binds requests to a
specific request service, such as an MDS, MGS, OSS, or LDLM. Services
may listen on multiple portals to ensure that high priority messages
are not queued behind many slow requests on another portal.</para>
<glossentry xml:id="ptlrpc">
<glossterm>PTLRPC</glossterm>
<glossdef>
- <para>An RPC protocol layered on LNET. This protocol deals with
+ <para>An RPC protocol layered on LNet. This protocol deals with
stateful servers and has exactly-once semantics and built in support
for recovery.</para>
</glossdef>
<glossentry xml:id="routing">
<glossterm>Routing</glossterm>
<glossdef>
- <para>LNET routing between different networks and LNDs.</para>
+ <para>LNet routing between different networks and LNDs.</para>
</glossdef>
</glossentry>
<glossentry xml:id="rpc">
</listitem>
<listitem>
<para><emphasis>(Optional)</emphasis>
- <emphasis role="bold">Configure Lustre networking (LNET).</emphasis></para>
- <para>See <xref linkend="configuringlnet"/> - Describes how to configure LNET if the default
- configuration is not sufficient. By default, LNET will use the first TCP/IP interface it
- discovers on a system. LNET configuration is required if you are using InfiniBand or
+ <emphasis role="bold">Configure Lustre Networking (LNet).</emphasis></para>
+ <para>See <xref linkend="configuringlnet"/> - Describes how to configure LNet if the default
+ configuration is not sufficient. By default, LNet will use the first TCP/IP interface it
+ discovers on a system. LNet configuration is required if you are using InfiniBand or
multiple Ethernet interfaces.</para>
</listitem>
<listitem>
<listitem>
<para>
<emphasis role="bold">
- <emphasis role="italic">Lustre LNET network driver (LND)</emphasis>
+ <emphasis role="italic">Lustre LNet network driver (LND)</emphasis>
</emphasis>. The Lustre LNDs provided with the Lustre software are
- listed in the table below. For more information about Lustre LNET,
+ listed in the table below. For more information about Lustre LNet,
see
<xref xmlns:xlink="http://www.w3.org/1999/xlink"
linkend="understandinglustrenetworking" />.</para>
</orderedlist>
</listitem>
</orderedlist>
- <para>To configure LNET, go to
+ <para>To configure LNet, go to
<xref linkend="configuringlnet" />. If default settings will be used for
- LNET, go to
+ LNet, go to
<xref linkend="configuringlustre" />.</para>
</section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetselftest">
- <title xml:id="lnetselftest.title">Testing Lustre Network Performance (LNET Self-Test)</title>
- <para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNET
- self-test, which is used by site administrators to confirm that Lustre networking (LNET) has
+ <title xml:id="lnetselftest.title">Testing Lustre Network Performance (LNet Self-Test)</title>
+ <para><anchor xml:id="dbdoclet.50438223_36273" xreflabel=""/>This chapter describes the LNet
+ self-test, which is used by site administrators to confirm that Lustre Networking (LNet) has
been properly installed and configured, and that underlying network software and hardware are
performing according to expectations. The chapter includes:</para>
<itemizedlist>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438223_91742">
- <title><indexterm><primary>LNET</primary><secondary>self-test</secondary></indexterm>
-LNET Self-Test Overview</title>
- <para>LNET self-test is a kernel module that runs over LNET and the Lustre network drivers (LNDs). It is designed to:</para>
+ <title><indexterm><primary>LNet</primary><secondary>self-test</secondary></indexterm>
+LNet Self-Test Overview</title>
+ <para>LNet self-test is a kernel module that runs over LNet and the Lustre network drivers (LNDs). It is designed to:</para>
<itemizedlist>
<listitem>
<para>Test the connection ability of the Lustre network</para>
<para>Test performance of the Lustre network</para>
</listitem>
</itemizedlist>
- <para>After you have obtained performance results for your Lustre network, refer to <xref linkend="lustretuning"/> for information about parameters that can be used to tune LNET for optimum performance.</para>
+ <para>After you have obtained performance results for your Lustre network, refer to <xref linkend="lustretuning"/> for information about parameters that can be used to tune LNet for optimum performance.</para>
<note>
- <para>Apart from the performance impact, LNET self-test is invisible to the Lustre file
+ <para>Apart from the performance impact, LNet self-test is invisible to the Lustre file
system.</para>
</note>
- <para>An LNET self-test cluster includes two types of nodes:</para>
+ <para>An LNet self-test cluster includes two types of nodes:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">Console node</emphasis> - A node used to control and monitor an LNET self-test cluster. The console node serves as the user interface of the LNET self-test system and can be any node in the test cluster. All self-test commands are entered from the console node. From the console node, a user can control and monitor the status of the entire LNET self-test cluster (session). The console node is exclusive in that a user cannot control two different sessions from one console node.</para>
+ <para><emphasis role="bold">Console node</emphasis> - A node used to control and monitor an LNet self-test cluster. The console node serves as the user interface of the LNet self-test system and can be any node in the test cluster. All self-test commands are entered from the console node. From the console node, a user can control and monitor the status of the entire LNet self-test cluster (session). The console node is exclusive in that a user cannot control two different sessions from one console node.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Test nodes</emphasis> - The nodes on which the tests are run. Test nodes are controlled by the user from the console node; the user does not need to log into them directly.</para>
</listitem>
</itemizedlist>
- <para>LNET self-test has two user utilities:</para>
+ <para>LNet self-test has two user utilities:</para>
<itemizedlist>
<listitem>
<para><emphasis role="bold">
<listitem>
<para><emphasis role="bold">
<literal>lstclient</literal>
- </emphasis> - The userspace LNET self-test program (run on a <emphasis>test node</emphasis>). The <literal>lstclient</literal> utility is linked with userspace LNDs and LNET. This utility is not needed if only kernel space LNET and LNDs are used.</para>
+ </emphasis> - The userspace LNet self-test program (run on a <emphasis>test node</emphasis>). The <literal>lstclient</literal> utility is linked with userspace LNDs and LNet. This utility is not needed if only kernel space LNet and LNDs are used.</para>
</listitem>
</itemizedlist>
<note>
</note>
<section remap="h3">
<title>Prerequisites</title>
- <para>To run LNET self-test, these modules must be loaded on both <emphasis>console nodes</emphasis> and <emphasis>test nodes</emphasis>:</para>
+ <para>To run LNet self-test, these modules must be loaded on both <emphasis>console nodes</emphasis> and <emphasis>test nodes</emphasis>:</para>
<itemizedlist>
<listitem>
<para><literal>libcfs</literal></para>
</itemizedlist>
<para>To load the required modules, run:</para>
<screen>modprobe lnet_selftest </screen>
- <para>This command recursively loads the modules on which LNET self-test depends.</para>
+ <para>This command recursively loads the modules on which LNet self-test depends.</para>
<note>
<para>While the <emphasis>console node</emphasis> and <emphasis>test nodes</emphasis> require all the prerequisite modules to be loaded, userspace test nodes do not require these modules.</para>
</note>
</section>
</section>
<section xml:id="dbdoclet.50438223_48138">
- <title>Using LNET Self-Test</title>
- <para>This section describes how to create and run an LNET self-test. The examples shown are for a test that simulates the traffic pattern of a set of Lustre servers on a TCP network accessed by Lustre clients on an InfiniBand network connected via LNET routers. In this example, half the clients are reading and half the clients are writing.</para>
+ <title>Using LNet Self-Test</title>
+ <para>This section describes how to create and run an LNet self-test. The examples shown are for a test that simulates the traffic pattern of a set of Lustre servers on a TCP network accessed by Lustre clients on an InfiniBand network connected via LNet routers. In this example, half the clients are reading and half the clients are writing.</para>
<section remap="h3">
<title>Creating a Session</title>
<para>A <emphasis>session</emphasis> is a set of processes that run on a <emphasis>test node</emphasis>. Only one session can be run at a time on a test node to ensure that the session has exclusive use of the node. The console node is used to create, change or destroy a session (<literal>new_session</literal>, <literal>end_session</literal>, <literal>show_session</literal>). For more about session parameters, see <xref linkend="dbdoclet.50438223_91247"/>.</para>
</section>
<section remap="h3">
<title>Setting Up Groups</title>
- <para>A <emphasis>group</emphasis> is a named collection of nodes. Any number of groups can exist in a single LNET self-test session. Group membership is not restricted in that a <emphasis>test node</emphasis> can be included in any number of groups.</para>
+ <para>A <emphasis>group</emphasis> is a named collection of nodes. Any number of groups can exist in a single LNet self-test session. Group membership is not restricted in that a <emphasis>test node</emphasis> can be included in any number of groups.</para>
<para>Each node in a group has a rank, determined by the order in which it was added to the group. The rank is used to establish test traffic patterns.</para>
<para>A user can only control nodes in his/her session. To allocate nodes to the session, the user needs to add nodes to a group (of the session). All nodes in a group can be referenced by the group name. A node can be allocated to multiple groups of a session.</para>
<para>In the following example, three groups are established on a console node:</para>
<para>These three groups include:</para>
<itemizedlist>
<listitem>
- <para>Nodes that will function as 'servers' to be accessed by 'clients' during the LNET self-test session</para>
+ <para>Nodes that will function as 'servers' to be accessed by 'clients' during the LNet self-test session</para>
</listitem>
<listitem>
<para>Nodes that will function as 'clients' that will simulate <emphasis>reading</emphasis> data from the 'servers'</para>
</section>
<section remap="h3">
<title>Sample Script</title>
- <para>This sample LNET self-test script simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNET routers). In this example, half the clients are reading and half the clients are writing.</para>
+ <para>This sample LNet self-test script simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an InfiniBand network (connected via LNet routers). In this example, half the clients are reading and half the clients are writing.</para>
<para>Run this script on the console node:</para>
<screen>#!/bin/bash
export LST_SESSION=$$
</section>
</section>
<section xml:id="dbdoclet.50438223_27277">
- <title>LNET Self-Test Command Reference</title>
- <para>The LNET self-test (<literal>lst</literal>) utility is used to issue LNET self-test commands. The <literal>lst</literal> utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific.</para>
+ <title>LNet Self-Test Command Reference</title>
+ <para>The LNet self-test (<literal>lst</literal>) utility is used to issue LNet self-test commands. The <literal>lst</literal> utility takes a number of command line arguments. The first argument is the command name and subsequent arguments are command-specific.</para>
<section xml:id="dbdoclet.50438223_91247">
<title>Session Commands</title>
<para>This section describes <literal>lst</literal> session commands.</para>
<para> <literal><replaceable>NIDs</replaceable></literal></para>
</entry>
<entry>
- <para>A string that may be expanded to include one or more LNET NIDs.</para>
+ <para>A string that may be expanded to include one or more LNet NIDs.</para>
</entry>
</row>
</tbody>
<literal>--server_mode</literal>
</entry>
<entry>
- <para> When included, forces LNET to behave as a server, such as starting an acceptor if the underlying NID needs it or using privileged ports. Only root is allowed to use the <literal>--server_mode</literal> option.</para>
+ <para> When included, forces LNet to behave as a server, such as starting an acceptor if the underlying NID needs it or using privileged ports. Only root is allowed to use the <literal>--server_mode</literal> option.</para>
</entry>
</row>
</tbody>
<para>where servers is the name of a test group created by <literal>lst add_group</literal></para>
<para>Specifying a <literal><replaceable>NID</replaceable></literal> range (<literal><replaceable>NIDs</replaceable></literal>) causes statistics to be gathered for selected nodes. For example:</para>
<screen>$ lst stat 192.168.0.[1-100/2]@tcp</screen>
- <para>Only LNET performance statistics are available. By default, all statistics
+ <para>Only LNet performance statistics are available. By default, all statistics
information is displayed. Users can specify additional information with these options.</para>
<para><literal>show_error [--session] [<replaceable>group</replaceable>|<replaceable>NIDs</replaceable>]... </literal></para>
<para>Lists the number of failed RPCs on test nodes.</para>
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetconfigurationapi">
<title xml:id="lnetconfigurationapi.title">LNet Configuration C-API</title>
- <para>This section describes the LNET Configuration C-API library. This
- API allows the developer to programatically configure LNET. It provides
- APIs to add, delete and show LNET configuration items listed below. The
+ <para>This section describes the LNet Configuration C-API library. This
+ API allows the developer to programatically configure LNet. It provides
+ APIs to add, delete and show LNet configuration items listed below. The
API utilizes IOCTL to communicate with the kernel. Changes take effect
- immediately and do not require restarting LNET. API calls are
+ immediately and do not require restarting LNet. API calls are
synchronous</para>
<para>
<itemizedlist>
</para>
<section remap="h5">
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>capi general information</secondary>
</indexterm>General API Information</title>
<para/>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>capi return code</secondary>
</indexterm>API Return Code</title>
<screen>LUSTRE_CFG_RC_NO_ERR 0
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>capi input params</secondary>
</indexterm>API Common Input Parameters</title>
<para>All APIs take as input a sequence number. This is a number
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>capi output params</secondary>
</indexterm>API Common Output Parameters</title>
<para/>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>cyaml</secondary>
</indexterm>Internal YAML Representation (cYAML)</title>
<para>Once a YAML block is parsed it needs to be stored
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>error block</secondary>
</indexterm>Error Block</title>
<para>All APIs return a cYAML error block. This error block has
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>show block</secondary>
</indexterm>Show Block</title>
<para>All Show APIs return a cYAML show block. This show block
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>show block</secondary>
- </indexterm>The LNET Configuration C-API</title>
+ </indexterm>The LNet Configuration C-API</title>
<para/>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_config_ni_system</secondary>
</indexterm>Configuring LNet</title>
<para/>
<screen>
/*
* lustre_lnet_config_ni_system
- * Initialize/Uninitialize the lnet NI system.
+ * Initialize/Uninitialize the LNet NI system.
*
* up - whether to init or uninit the system
* load_ni_from_mod - load NI from mod params.
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_enable_routing</secondary>
</indexterm>Enabling and Disabling Routing</title>
<para/>
messages not destined to this node are dropped. </para>
<para><emphasis role="bold">Enabling Routing on an already enabled
node, or vice versa</emphasis></para>
- <para>In both these cases the LNET Kernel module ignores this request. </para>
+ <para>In both these cases the LNet Kernel module ignores this request. </para>
<para><emphasis role="bold">Return Value</emphasis></para>
<para>-ENOMEM: if there is no memory to allocate buffer pools</para>
<para>0: if success</para>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_config_route</secondary>
</indexterm>Adding Routes</title>
<para/>
<para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
<para>IOC_LIBCFS_ADD_ROUTE</para>
<para><emphasis role="bold">Description:</emphasis></para>
- <para>The LNET Kernel module adds this route to the list of
+ <para>The LNet Kernel module adds this route to the list of
existing routes, if one doesn't already exist. If hop parameter is
not specified (IE: -1) then the hop count is set to 1. If the
priority parameter is not specified (IE: -1) then the priority is
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_del_route</secondary>
</indexterm>Deleting Routes</title>
<para/>
<para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
<para>IOC_LIBCFS_DEL_ROUTE</para>
<para><emphasis role="bold">Description:</emphasis></para>
- <para>LNET will remove the route which matches the network and gateway passed in. If
+ <para>LNet will remove the route which matches the network and gateway passed in. If
no route matches, then the operation fails with an appropriate error number.</para>
<para><emphasis role="bold">Return Value</emphasis></para>
<para>-ENOENT: if the entry being deleted doesn't exist</para>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_show_route</secondary>
</indexterm>Showing Routes</title>
<para/>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_config_net</secondary>
</indexterm>Adding a Network Interface</title>
<para/>
peer timeout, peer credits, peer buffer credits and credits. The
CPU affinity of the network interface being added can also be
specified. These parameters become
- network specific under Dynamic LNET Configuration (DLC), as
+ network specific under Dynamic LNet Configuration (DLC), as
opposed to being per LND as it was previously.</para>
<para>If an already existing network is added the request is ignored.</para>
<para><emphasis role="bold">Return Value</emphasis></para>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_del_net</secondary>
</indexterm>Deleting a Network Interface</title>
<para/>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_show_net</secondary>
</indexterm>Showing Network Interfaces</title>
<para/>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_config_buf</secondary>
</indexterm>Adjusting Router Buffer Pools</title>
<para/>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_show_buf</secondary>
</indexterm>Showing Routing information</title>
<para/>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_lnet_show stats</secondary>
- </indexterm>Showing LNET Traffic Statistics</title>
+ </indexterm>Showing LNet Traffic Statistics</title>
<para/>
<screen>/*
* lustre_lnet_show_stats
- * Shows internal LNET statistics. This is useful to display the
- * current LNET activity, such as number of messages route, etc
+ * Shows internal LNet statistics. This is useful to display the
+ * current LNet activity, such as number of messages route, etc
*
* seq_no - sequence number of the command
* show_rc - YAML structure of the resultant show
<para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
<para>IOC_LIBCFS_GET_LNET_STATS</para>
<para><emphasis role="bold">Description:</emphasis></para>
- <para>This API returns a cYAML block describing the LNET traffic
- statistics. Statistics are continuously incremented by LNET while
+ <para>This API returns a cYAML block describing the LNet traffic
+ statistics. Statistics are continuously incremented by LNet while
it's alive. This API retuns the statistics at the time of the API
call. The statistics include the following</para>
<para>
</section>
<section>
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>lustre_yaml</secondary>
</indexterm>Adding/Deleting/Showing Parameters through a YAML Block</title>
<para/>
packet inspection tool that allows debugging of information that was
sent between the various Lustre nodes. This tool is built on top of
<literal>tcpdump</literal> and can read packet dumps generated by
- it. There are plug-ins available to dissassemble the LNET and
+ it. There are plug-ins available to dissassemble the LNet and
Lustre protocols. They are located within the <link
xl:href="http://git.hpdd.intel.com/">Lustre git repository</link>
under <literal>lustre/contrib/wireshark/</literal>. Installation
</emphasis></para>
</entry>
<entry>
- <para> Behaves similarly to <literal>CERROR()</literal>, but prints error messages for LNET if <literal>D_NETERR</literal> is set in the <literal>debug</literal> mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited.</para>
+ <para> Behaves similarly to <literal>CERROR()</literal>, but prints error messages for LNet if <literal>D_NETERR</literal> is set in the <literal>debug</literal> mask. This is appropriate for serious networking errors. Messages printed to the console are rate-limited.</para>
</entry>
</row>
<row>
<section remap="h3">
<title>Accessing the <literal>ptlrpc</literal> Request History</title>
<para>Each service maintains a request history, which can be useful for first occurrence troubleshooting.</para>
- <para><literal>ptlrpc</literal> is an RPC protocol layered on LNET that deals with stateful servers and has semantics and built-in support for recovery.</para>
+ <para><literal>ptlrpc</literal> is an RPC protocol layered on LNet that deals with stateful servers and has semantics and built-in support for recovery.</para>
<para>The ptlrpc request history works as follows:</para>
<orderedlist>
<listitem>
<para>To change a server NID:</para>
<orderedlist>
<listitem>
- <para>Update the LNET configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs is correct. Use <literal>lctl list_nids</literal> to view the list of server NIDS.</para>
+ <para>Update the LNet configuration in the <literal>/etc/modprobe.conf</literal> file so the list of server NIDs is correct. Use <literal>lctl list_nids</literal> to view the list of server NIDS.</para>
<para>The <literal>lctl list_nids</literal> command indicates which network(s) are
configured to work with the Lustre file system.</para>
</listitem>
<para>In the course of administering a Lustre file system, you may need to determine which
machine is serving a specific OST. It is not as simple as identifying the machine’s IP
address, as IP is only one of several networking protocols that the Lustre software uses and,
- as such, LNET does not use IP addresses as node identifiers, but NIDs instead. To identify the
+ as such, LNet does not use IP addresses as node identifiers, but NIDs instead. To identify the
NID that is serving a specific OST, run one of the following commands on a client (you do not
need to be a root user):
<screen>client$ lctl get_param osc.<replaceable>fsname</replaceable>-OST<replaceable>number</replaceable>*.ost_conn_uuid</screen>
is described using a dash to separate the range, for example,
<literal>192.168.20.[0-255]@tcp</literal>.</para>
- <para>The range must be contiguous. The full LNET definition for a
+ <para>The range must be contiguous. The full LNet definition for a
nidlist is as follows:</para>
<screen>
</section>
<section xml:id="dbdoclet.50438194_88217">
<title>Listing Parameters</title>
- <para>To list Lustre or LNET parameters that are available to set, use
+ <para>To list Lustre or LNet parameters that are available to set, use
the
<literal>lctl list_param</literal> command. For example:</para>
<screen>
<literal>--mgsnode=</literal> or
<literal>--servicenode=</literal>).</para>
<para>To display the NIDs of all servers in networks configured to work
- with the Lustre file system, run (while LNET is running):</para>
+ with the Lustre file system, run (while LNet is running):</para>
<screen>
lctl list_nids
</screen>
messages or enable printing of <literal>D_NETERROR</literal> messages to the console
using:<screen>lctl set_param printk=+neterror</screen></para>
<para>Congested routers can be a source of spurious LND timeouts. To avoid this
- situation, increase the number of LNET router buffers to reduce back-pressure and/or
+ situation, increase the number of LNet router buffers to reduce back-pressure and/or
increase LND timeouts on all nodes on all connected networks. Also consider increasing
- the total number of LNET router nodes in the system so that the aggregate router
+ the total number of LNet router nodes in the system so that the aggregate router
bandwidth matches the aggregate server bandwidth.</para>
</listitem>
<listitem>
<section remap="h3">
<title><indexterm>
<primary>proc</primary>
- <secondary>LNET</secondary>
+ <secondary>LNet</secondary>
</indexterm><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>proc</secondary>
- </indexterm>Monitoring LNET</title>
- <para>LNET information is located in <literal>/proc/sys/lnet</literal> in these files:<itemizedlist>
+ </indexterm>Monitoring LNet</title>
+ <para>LNet information is located in <literal>/proc/sys/lnet</literal> in these files:<itemizedlist>
<listitem>
<para><literal>peers</literal> - Shows all NIDs known to this node and provides
information on the queue state.</para>
</tgroup>
</informaltable>
<para>Credits are initialized to allow a certain number of operations (in the example
- above the table, eight as shown in the <literal>max</literal> column. LNET keeps track
+ above the table, eight as shown in the <literal>max</literal> column. LNet keeps track
of the minimum number of credits ever seen over time showing the peak congestion that
has occurred during the time monitored. Fewer available credits indicates a more
congested resource. </para>
credits (<literal>rtr/tx</literal>) that is less than <literal>max</literal> indicates
operations are in progress. If the ratio <literal>rtr/tx</literal> is greater than
<literal>max</literal>, operations are blocking.</para>
- <para>LNET also limits concurrent sends and number of router buffers allocated to a single
+ <para>LNet also limits concurrent sends and number of router buffers allocated to a single
peer so that no peer can occupy all these resources.</para>
</listitem>
<listitem>
use</literal> error and reject to start the operation. This is caused by a portmap service
(often NFS locking) that starts before the Lustre file system and binds to the default port
988. You must have port 988 open from firewall or IP tables for incoming connections on the
- client, OSS, and MDS nodes. LNET will create three outgoing connections on available,
+ client, OSS, and MDS nodes. LNet will create three outgoing connections on available,
reserved ports to each client-server pair, starting with 1023, 1022 and 1021.</para>
<para>Unfortunately, you cannot set sunprc to avoid port 988. If you receive this error, do the following:</para>
<itemizedlist>
</listitem>
<listitem>
<para>Use a port other than 988 for the Lustre file system. This is configured in
- <literal>/etc/modprobe.d/lustre.conf</literal> as an option to the LNET module. For
+ <literal>/etc/modprobe.d/lustre.conf</literal> as an option to the LNet module. For
example:</para>
<screen>options lnet accept_port=988</screen>
</listitem>
<section xml:id="dbdoclet.50438272_73839">
<title>
<indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>tuning</secondary>
</indexterm>
<indexterm>
<primary>tuning</primary>
- <secondary>LNET</secondary>
- </indexterm>Tuning LNET Parameters</title>
- <para>This section describes LNET tunables, the use of which may be
+ <secondary>LNet</secondary>
+ </indexterm>Tuning LNet Parameters</title>
+ <para>This section describes LNet tunables, the use of which may be
necessary on some systems to improve performance. To test the performance
of your Lustre network, see
<xref linkend='lnetselftest' />.</para>
<para>Lustre software release 2.3 and beyond provide enhanced network
interface control. The enhancement means that an administrator can bind
an interface to one or more CPU partitions. Bindings are specified as
- options to the LNET modules. For more information on specifying module
+ options to the LNet modules. For more information on specifying module
options, see
<xref linkend="dbdoclet.50438293_15350" /></para>
<para>For example,
<para>Network interface (NI) credits are shared across all CPU partitions
(CPT). For example, if a machine has four CPTs and the number of NI
credits is 512, then each partition has 128 credits. If a large number of
- CPTs exist on the system, LNET checks and validates the NI credits for
+ CPTs exist on the system, LNet checks and validates the NI credits for
each CPT to ensure each CPT has a workable number of credits. For
example, if a machine has 16 CPTs and the number of NI credits is 256,
then each partition only has 16 credits. 16 NI credits is low and could
- negatively impact performance. As a result, LNET automatically adjusts
+ negatively impact performance. As a result, LNet automatically adjusts
the credits to 8*
<literal>peer_credits</literal>(
<literal>peer_credits</literal> is 8 by default), so each partition has 64
<para>Increasing the number of
<literal>credits</literal>/
<literal>peer_credits</literal> can improve the performance of high
- latency networks (at the cost of consuming more memory) by enabling LNET
+ latency networks (at the cost of consuming more memory) by enabling LNet
to send more inflight messages to a specific network/peer and keep the
pipeline saturated.</para>
<para>An administrator can modify the NI credit count using
ko2iblnd credits=256
</screen>
<note condition="l23">
- <para>In Lustre software release 2.3 and beyond, LNET may revalidate
+ <para>In Lustre software release 2.3 and beyond, LNet may revalidate
the NI credits, so the administrator's request may not persist.</para>
</note>
</section>
<primary>tuning</primary>
<secondary>router buffers</secondary>
</indexterm>Router Buffers</title>
- <para>When a node is set up as an LNET router, three pools of buffers are
+ <para>When a node is set up as an LNet router, three pools of buffers are
allocated: tiny, small and large. These pools are allocated per CPU
partition and are used to buffer messages that arrive at the router to be
forwarded to the next hop. The three different buffer sizes accommodate
</listitem>
</itemizedlist>
<para>The default setting for router buffers typically results in
- acceptable performance. LNET automatically sets a default value to reduce
+ acceptable performance. LNet automatically sets a default value to reduce
the likelihood of resource starvation. The size of a router buffer can be
modified as shown in the example below. In this example, the size of the
large buffer is modified using the
lnet large_router_buffers=8192
</screen>
<note condition="l23">
- <para>In Lustre software release 2.3 and beyond, LNET may revalidate
+ <para>In Lustre software release 2.3 and beyond, LNet may revalidate
the router buffer setting, so the administrator's request may not
persist.</para>
</note>
<primary>tuning</primary>
<secondary>portal round-robin</secondary>
</indexterm>Portal Round-Robin</title>
- <para>Portal round-robin defines the policy LNET applies to deliver
+ <para>Portal round-robin defines the policy LNet applies to deliver
events and messages to the upper layers. The upper layers are PLRPC
- service or LNET selftest.</para>
- <para>If portal round-robin is disabled, LNET will deliver messages to
+ service or LNet selftest.</para>
+ <para>If portal round-robin is disabled, LNet will deliver messages to
CPTs based on a hash of the source NID. Hence, all messages from a
specific peer will be handled by the same CPT. This can reduce data
traffic between CPUs. However, for some workloads, this behavior may
result in poorly balancing loads across the CPU.</para>
- <para>If portal round-robin is enabled, LNET will round-robin incoming
+ <para>If portal round-robin is enabled, LNet will round-robin incoming
events across all CPTs. This may balance load better across the CPU but
can incur a cross CPU overhead.</para>
<para>The current policy can be changed by an administrator with
</itemizedlist>
</section>
<section>
- <title>LNET Peer Health</title>
+ <title>LNet Peer Health</title>
<para>Two options are available to help determine peer health:
<itemizedlist>
<listitem>
<literal>peer_timeout</literal> is set to
<literal>180sec</literal>, an aliveness query is sent to the peer
every 180 seconds. This feature only takes effect if the node is
- configured as an LNET router.</para>
+ configured as an LNet router.</para>
<para>In a routed environment, the
<literal>peer_timeout</literal> feature should always be on (set to a
value in seconds) on routers. If the router checker has been enabled,
all the routers corresponding to the NIDs identified in the routes
parameter setting on the node to determine the status of each router
interface. The default setting is 1. (For more information about the
- LNET routes parameter, see
+ LNet routes parameter, see
<xref xmlns:xlink="http://www.w3.org/1999/xlink"
linkend="dbdoclet.50438216_71227" /></para>
<para>A router is considered down if any of its NIDs are down. For
<tertiary>first in, first out (FIFO) policy</tertiary>
</indexterm>First In, First Out (FIFO) policy</title>
<para>The first in, first out (FIFO) policy handles RPCs in a service in
- the same order as they arrive from the LNET layer, so no special
+ the same order as they arrive from the LNet layer, so no special
processing takes place to modify the RPC handling stream. FIFO is the
default policy for all types of RPCs on all PTLRPC services, and is
always enabled irrespective of the state of other policies, so that it
</screen>
<para>The format of '
<replaceable>nidlist</replaceable>' argument is the same as the
- format when configuring LNET route. The '
+ format when configuring LNet route. The '
<replaceable>rate</replaceable>' argument is the RPC rate of the
rule, means the upper limit number of requests per second.</para>
<para>Following commands are valid. Please note that a newly started
<?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="managinglnet">
- <title xml:id="managinglnet.title">Managing Lustre Networking (LNET)</title>
- <para>This chapter describes some tools for managing Lustre networking (LNET) and includes the
+ <title xml:id="managinglnet.title">Managing Lustre Networking (LNet)</title>
+ <para>This chapter describes some tools for managing Lustre networking (LNet) and includes the
following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438203_51732">
- <title><indexterm><primary>LNET</primary><secondary>management</secondary></indexterm>
+ <title><indexterm><primary>LNet</primary><secondary>management</secondary></indexterm>
Updating the Health Status of a Peer or Router</title>
<para>There are two mechanisms to update the health status of a peer or a router:</para>
<itemizedlist>
<listitem>
- <para>LNET can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set <literal>auto_down</literal> and if desired <literal>check_routers_before_use</literal>. This initial check may cause a pause equal to <literal>router_ping_timeout</literal> at system startup, if there are dead routers in the system.</para>
+ <para>LNet can actively check health status of all routers and mark them as dead or alive automatically. By default, this is off. To enable it set <literal>auto_down</literal> and if desired <literal>check_routers_before_use</literal>. This initial check may cause a pause equal to <literal>router_ping_timeout</literal> at system startup, if there are dead routers in the system.</para>
</listitem>
<listitem>
- <para>When there is a communication error, all LNDs notify LNET that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNET module parameter <literal>auto_down</literal> to <literal>0</literal>, LNET ignores all such peer-down notifications.</para>
+ <para>When there is a communication error, all LNDs notify LNet that the peer (not necessarily a router) is down. This mechanism is always on, and there is no parameter to turn it off. However, if you set the LNet module parameter <literal>auto_down</literal> to <literal>0</literal>, LNet ignores all such peer-down notifications.</para>
</listitem>
</itemizedlist>
<para>Several key differences in both mechanisms:</para>
</itemizedlist>
</section>
<section xml:id="dbdoclet.50438203_48703">
- <title><indexterm><primary>LNET</primary><secondary>starting/stopping</secondary></indexterm>Starting and Stopping LNET</title>
- <para>The Lustre software automatically starts and stops LNET, but it can also be manually
+ <title><indexterm><primary>LNet</primary><secondary>starting/stopping</secondary></indexterm>Starting and Stopping LNet</title>
+ <para>The Lustre software automatically starts and stops LNet, but it can also be manually
started in a standalone manner. This is particularly useful to verify that your networking
setup is working correctly before you attempt to start the Lustre file system.</para>
<section remap="h3">
- <title>Starting LNET</title>
- <para>To start LNET, run:</para>
+ <title>Starting LNet</title>
+ <para>To start LNet, run:</para>
<screen>$ modprobe lnet
$ lctl network up</screen>
<para>To see the list of local NIDs, run:</para>
</section>
</section>
<section remap="h3">
- <title>Stopping LNET</title>
- <para>Before the LNET modules can be removed, LNET references must be removed. In general,
+ <title>Stopping LNet</title>
+ <para>Before the LNet modules can be removed, LNet references must be removed. In general,
these references are removed automatically when the Lustre file system is shut down, but for
- standalone routers, an explicit step is needed to stop LNET. Run:</para>
+ standalone routers, an explicit step is needed to stop LNet. Run:</para>
<screen>lctl network unconfigure</screen>
<note>
<para>Attempting to remove Lustre modules prior to stopping the network may result in a
- crash or an LNET hang. If this occurs, the node must be rebooted (in most cases). Make
+ crash or an LNet hang. If this occurs, the node must be rebooted (in most cases). Make
sure that the Lustre network and Lustre file system are stopped prior to unloading the
modules. Be extremely careful using <literal>rmmod -f</literal>.</para>
</note>
- <para>To unconfigure the LNET network, run:</para>
+ <para>To unconfigure the LNet network, run:</para>
<screen>modprobe -r <replaceable>lnd_and_lnet_modules</replaceable></screen>
<note>
<para>
</section>
</section>
<section xml:id="dbdoclet.50438203_82542">
- <title><indexterm><primary>LNET</primary><secondary>multi-rail configuration</secondary></indexterm>Multi-Rail Configurations with LNET</title>
+ <title><indexterm><primary>LNet</primary><secondary>multi-rail configuration</secondary></indexterm>Multi-Rail Configurations with LNet</title>
<para>To aggregate bandwidth across both rails of a dual-rail IB cluster (o2iblnd) <footnote>
<para>Multi-rail configurations are only supported by o2iblnd; other IB LNDs do not support multiple interfaces.</para>
- </footnote> using LNET, consider these points:</para>
+ </footnote> using LNet, consider these points:</para>
<itemizedlist>
<listitem>
- <para>LNET can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
+ <para>LNet can work with multiple rails, however, it does not load balance across them. The actual rail used for any communication is determined by the peer NID.</para>
</listitem>
<listitem>
- <para>Multi-rail LNET configurations do not provide an additional level of network fault
+ <para>Multi-rail LNet configurations do not provide an additional level of network fault
tolerance. The configurations described below are for bandwidth aggregation only. </para>
</listitem>
<listitem>
<para>Fewest hops (to minimize routing), and</para>
</listitem>
<listitem>
- <para>Appears first in the "<literal>networks</literal>" or "<literal>ip2nets</literal>" LNET configuration strings</para>
+ <para>Appears first in the "<literal>networks</literal>" or "<literal>ip2nets</literal>" LNet configuration strings</para>
</listitem>
</itemizedlist>
</listitem>
</section>
<section xml:id="dbdoclet.50438203_78227">
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>InfiniBand load balancing</secondary>
</indexterm>Load Balancing with an InfiniBand<superscript>*</superscript> Network</title>
<para>A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one
InfiniBand HCA using OFED-based Infiniband ''o2ib'' drivers. Load
- balancing between the HCAs on the OSS is accomplished through LNET.</para>
+ balancing between the HCAs on the OSS is accomplished through LNet.</para>
<section remap="h3">
- <title><indexterm><primary>LNET</primary><secondary>lustre.conf</secondary></indexterm>Setting Up <literal>lustre.conf</literal> for Load Balancing</title>
- <para>To configure LNET for load balancing on clients and servers:</para>
+ <title><indexterm><primary>LNet</primary><secondary>lustre.conf</secondary></indexterm>Setting Up <literal>lustre.conf</literal> for Load Balancing</title>
+ <para>To configure LNet for load balancing on clients and servers:</para>
<orderedlist>
<listitem>
<para>Set the <literal>lustre.conf</literal> options.</para>
</section>
</section>
<section xml:id="managinglnet.configuringroutes" condition='l24'>
- <title><indexterm><primary>LNET</primary></indexterm>Dynamically Configuring LNET Routes</title>
+ <title><indexterm><primary>LNet</primary></indexterm>Dynamically Configuring LNet Routes</title>
<para>Two scripts are provided: <literal>lustre/scripts/lustre_routes_config</literal> and <literal>lustre/scripts/lustre_routes_conversion</literal>.</para>
- <para><literal>lustre_routes_config</literal> sets or cleans up LNET routes from the specified config file. <literal>/etc/sysconfig/lustre_routes.conf</literal> file can be used to automatically configure routes on LNET startup. </para>
+ <para><literal>lustre_routes_config</literal> sets or cleans up LNet routes from the specified config file. <literal>/etc/sysconfig/lustre_routes.conf</literal> file can be used to automatically configure routes on LNet startup. </para>
<para><literal>lustre_routes_conversion</literal> converts a legacy routes configuration file to the new syntax, which is parsed by <literal>lustre_routes_config</literal>.</para>
<section remap="h3">
- <title><indexterm><primary>LNET</primary></indexterm><literal>lustre_routes_config</literal></title>
+ <title><indexterm><primary>LNet</primary></indexterm><literal>lustre_routes_config</literal></title>
<para><literal>lustre_routes_config</literal> usage is as follows</para>
<screen><literal>lustre_routes_config [--setup|--cleanup|--dry-run|--verbose] <replaceable>config_file</replaceable></literal>
--setup: configure routes listed in config_file
--verbose: echo commands before they are executed </screen>
<para> The format of the file which is passed into the script is as follows: </para>
<para><literal><replaceable>network</replaceable>: { gateway: <replaceable>gateway</replaceable>@<replaceable>exit_network</replaceable> [hop: <replaceable>hop</replaceable>] [priority: <replaceable>priority</replaceable>] }</literal></para>
- <para> An LNET router is identified when its local NID appears within the list of routes. However, this can not be achieved by the use of this script, since the script only adds extra routes after the router is identified. To ensure that a router is identified correctly, make sure to add its local NID in the routes parameter in the modprobe lustre configuration file. See <xref linkend='dbdoclet.50438293_15350'/>.</para>
+ <para> An LNet router is identified when its local NID appears within the list of routes. However, this can not be achieved by the use of this script, since the script only adds extra routes after the router is identified. To ensure that a router is identified correctly, make sure to add its local NID in the routes parameter in the modprobe lustre configuration file. See <xref linkend='dbdoclet.50438293_15350'/>.</para>
</section>
<section remap="h3">
- <title><indexterm><primary>LNET</primary></indexterm><literal>lustre_routes_conversion</literal></title>
+ <title><indexterm><primary>LNet</primary></indexterm><literal>lustre_routes_conversion</literal></title>
<para><literal>lustre_routes_conversion</literal> usage is as follows:</para>
<screen><literal>lustre_routes_conversion <replaceable>legacy_file</replaceable> <replaceable>new_file</replaceable></literal></screen>
<para><literal>lustre_routes_conversion</literal> takes as a first parameter a file with routes configured as follows:</para>
<para>and appends each converted entry to the output file passed in as the second parameter to the script.</para>
</section>
<section remap="h3">
- <title><indexterm><primary>LNET</primary></indexterm><literal>Route Configuration Examples</literal></title>
- <para>Below is an example of a legacy LNET route configuration. A legacy configuration file can have multiple entries.</para>
+ <title><indexterm><primary>LNet</primary></indexterm><literal>Route Configuration Examples</literal></title>
+ <para>Below is an example of a legacy LNet route configuration. A legacy configuration file can have multiple entries.</para>
<screen><literal>tcp1 10.1.1.2@tcp0:1;
tcp2 10.1.1.3@tcp0:2;
tcp3 10.1.1.4@tcp0;</literal></screen>
- <para>Below is an example of the converted LNET route configuration. The following would be the result of the <literal>lustre_routes_conversion</literal> script, when run on the above legacy entries.</para>
+ <para>Below is an example of the converted LNet route configuration. The following would be the result of the <literal>lustre_routes_conversion</literal> script, when run on the above legacy entries.</para>
<screen><literal>tcp1: { gateway: 10.1.1.2@tcp0 priority: 1 }
tcp2: { gateway: 10.1.1.2@tcp0 priority: 2 }
tcp1: { gateway: 10.1.1.4@tcp0 }</literal></screen>
<para>The <literal>root_squash</literal> parameter specifies the UID and GID with which the root user accesses the Lustre file system.</para>
</listitem>
<listitem>
- <para>The <literal>nosquash_nids</literal> parameter specifies the set of clients to which root squash does not apply. LNET NID range syntax is used for this parameter (see the NID range syntax rules described in <xref linkend="dbdoclet.50438221_48757"/>). For example:</para>
+ <para>The <literal>nosquash_nids</literal> parameter specifies the set of clients to which root squash does not apply. LNet NID range syntax is used for this parameter (see the NID range syntax rules described in <xref linkend="dbdoclet.50438221_48757"/>). For example:</para>
</listitem>
</itemizedlist>
<screen>nosquash_nids=172.16.245.[0-255/2]@tcp</screen>
<para><literal>mkfs.lustre</literal> and <literal>tunefs.lustre</literal> do not perform parameter syntax checking. If the root squash parameters are incorrect, they are ignored on mount and the default values are used instead.</para>
</listitem>
<listitem>
- <para>Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <literal><decnum>:<decnum></literal>. The <literal>nosquash_nids</literal> parameter should follow LNET NID range list syntax.</para>
+ <para>Root squash parameters are parsed with rigorous syntax checking. The root_squash parameter should be specified as <literal><decnum>:<decnum></literal>. The <literal>nosquash_nids</literal> parameter should follow LNet NID range list syntax.</para>
</listitem>
</itemizedlist>
- <para>LNET NID range syntax:</para>
+ <para>LNet NID range syntax:</para>
<screen><nidlist> :== <nidrange> [ ' ' <nidrange> ]
<nidrange> :== <addrrange> '@' <net>
<addrrange> :== '*' |
</listitem>
</itemizedlist>
<para>Lustre networks and routing are configured and managed by specifying parameters to the
- Lustre networking (<literal>lnet</literal>) module in
+ Lustre Networking (<literal>lnet</literal>) module in
<literal>/etc/modprobe.d/lustre.conf</literal>.</para>
<para>To prepare to configure Lustre networking, complete the following steps:</para>
<orderedlist>
<listitem>
<para><emphasis role="bold">If routing is needed, identify the nodes to be used to route traffic between networks.</emphasis></para>
<para>If you are using multiple network types, then you will need a router. Any node with
- appropriate interfaces can route Lustre networking (LNET) traffic between different
+ appropriate interfaces can route Lustre networking (LNet) traffic between different
network hardware types or topologies --the node may be a server, a client, or a standalone
- router. LNET can route messages between different network types (such as
+ router. LNet can route messages between different network types (such as
TCP-to-InfiniBand) or across different topologies (such as bridging two InfiniBand or
TCP/IP networks). Routing will be configured in <xref linkend="configuringlnet"/>.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Identify the network interfaces to include
- in or exclude from LNET.</emphasis></para>
- <para>If not explicitly specified, LNET uses either the first available
+ in or exclude from LNet.</emphasis></para>
+ <para>If not explicitly specified, LNet uses either the first available
interface or a pre-defined default for a given network type. Interfaces
- that LNET should not use (such as an administrative network or
+ that LNet should not use (such as an administrative network or
IP-over-IB), can be excluded.</para>
<para>Network interfaces to be used or excluded will be specified using
the lnet kernel module parameters <literal>networks</literal> and
<para> <literal>network up|down|tcp|elan|myrinet</literal></para>
</entry>
<entry>
- <para> Starts or stops LNET, or selects a network type for other <literal>lctl</literal> LNET commands.</para>
+ <para> Starts or stops LNet, or selects a network type for other <literal>lctl</literal> LNet commands.</para>
</entry>
</row>
<row>
<para> <literal>list_nids</literal></para>
</entry>
<entry>
- <para> Prints all NIDs on the local node. LNET must be running.</para>
+ <para> Prints all NIDs on the local node. LNet must be running.</para>
</entry>
</row>
<row>
<para> <literal>ping <replaceable>nid</replaceable></literal></para>
</entry>
<entry>
- <para> Checks LNET connectivity via an LNET ping. This uses the fabric appropriate to the specified NID.</para>
+ <para> Checks LNet connectivity via an LNet ping. This uses the fabric appropriate to the specified NID.</para>
</entry>
</row>
<row>
<para> <literal>list_param [-F|-R] <replaceable>parameter</replaceable> <replaceable>[parameter ...]</replaceable></literal></para>
</entry>
<entry>
- <para> Lists the Lustre or LNET parameter name.</para>
+ <para> Lists the Lustre or LNet parameter name.</para>
<para> </para>
</entry>
</row>
<para> <literal>get_param [-n|-N|-F] <replaceable>parameter</replaceable> <replaceable>[parameter ...]</replaceable></literal></para>
</entry>
<entry>
- <para> Gets the value of a Lustre or LNET parameter from the specified path.</para>
+ <para> Gets the value of a Lustre or LNet parameter from the specified path.</para>
</entry>
</row>
<row>
<para> <literal>set_param [-n] <replaceable>parameter</replaceable>=<replaceable>value</replaceable></literal></para>
</entry>
<entry>
- <para> Sets the value of a Lustre or LNET parameter from the specified path.</para>
+ <para> Sets the value of a Lustre or LNet parameter from the specified path.</para>
</entry>
</row>
<row>
<section xml:id="dbdoclet.50438219_90218">
<title><indexterm><primary>lst</primary></indexterm>
lst</title>
- <para>The lst utility starts LNET self-test.</para>
+ <para>The lst utility starts LNet self-test.</para>
<section remap="h5">
<title>Synopsis</title>
<screen>lst</screen>
</section>
<section remap="h5">
<title>Description</title>
- <para>LNET self-test helps site administrators confirm that Lustre Networking (LNET) has been properly installed and configured. The self-test also confirms that LNET and the network software and hardware underlying it are performing as expected.</para>
- <para>Each LNET self-test runs in the context of a session. A node can be associated with only one session at a time, to ensure that the session has exclusive use of the nodes on which it is running. A session is create, controlled and monitored from a single node; this is referred to as the self-test console.</para>
+ <para>LNet self-test helps site administrators confirm that Lustre Networking (LNet) has been properly installed and configured. The self-test also confirms that LNet and the network software and hardware underlying it are performing as expected.</para>
+ <para>Each LNet self-test runs in the context of a session. A node can be associated with only one session at a time, to ensure that the session has exclusive use of the nodes on which it is running. A session is create, controlled and monitored from a single node; this is referred to as the self-test console.</para>
<para>Any node may act as the self-test console. Nodes are named and allocated to a self-test session in groups. This allows all nodes in a group to be referenced by a single name.</para>
<para>Test configurations are built by describing and running test batches. A test batch is a named collection of tests, with each test composed of a number of individual point-to-point tests running in parallel. These individual point-to-point tests are instantiated according to the test type, source group, target group and distribution specified when the test is added to the test batch.</para>
</section>
<section remap="h5">
<title>Modules</title>
- <para>To run LNET self-test, load these modules: libcfs, lnet, lnet_selftest and any one of the klnds (ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest, which recursively loads the modules on which lnet_selftest depends.</para>
- <para>There are two types of nodes for LNET self-test: the console node and test nodes. Both node types require all previously-specified modules to be loaded. (The userspace test node does not require these modules).</para>
+ <para>To run LNet self-test, load these modules: libcfs, lnet, lnet_selftest and any one of the klnds (ksocklnd, ko2iblnd...). To load all necessary modules, run modprobe lnet_selftest, which recursively loads the modules on which lnet_selftest depends.</para>
+ <para>There are two types of nodes for LNet self-test: the console node and test nodes. Both node types require all previously-specified modules to be loaded. (The userspace test node does not require these modules).</para>
<para>Test nodes can be in either kernel or in userspace. A console user can invite a kernel test node to join the test session by running lst add_group NID, but the user cannot actively add a userspace test node to the test session. However, the console user can passively accept a test node to the test session while the test node runs lst client to connect to the console.</para>
</section>
<section remap="h5">
<title>Utilities</title>
- <para>LNET self-test includes two user utilities, lst and lstclient.</para>
+ <para>LNet self-test includes two user utilities, lst and lstclient.</para>
<para>lst is the user interface for the self-test console (run on the console node). It provides a list of commands to control the entire test system, such as create session, create test groups, etc.</para>
- <para>lstclient is the userspace self-test program which is linked with userspace LNDs and LNET. A user can invoke lstclient to join a self-test session:</para>
+ <para>lstclient is the userspace self-test program which is linked with userspace LNDs and LNet. A user can invoke lstclient to join a self-test session:</para>
<screen>lstclient -sesid CONSOLE_NID group NAME</screen>
</section>
<section remap="h5">
<title>Example Script</title>
- <para>This is a sample LNET self-test script which simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an IB network (connected via LNET routers), with half the clients reading and half the clients writing.</para>
+ <para>This is a sample LNet self-test script which simulates the traffic pattern of a set of Lustre servers on a TCP network, accessed by Lustre clients on an IB network (connected via LNet routers), with half the clients reading and half the clients writing.</para>
<screen>#!/bin/bash
export LST_SESSION=$$
lst new_session read/write
<section xml:id="dbdoclet.50438219_54734">
<title><indexterm><primary>lustre_rmmod.sh</primary></indexterm>
lustre_rmmod.sh</title>
- <para>The lustre_rmmod.sh utility removes all Lustre and LNET modules (assuming no Lustre services are running). It is located in /usr/bin.</para>
+ <para>The lustre_rmmod.sh utility removes all Lustre and LNet modules (assuming no Lustre services are running). It is located in /usr/bin.</para>
<note>
<para>The lustre_rmmod.sh utility does not work if Lustre modules are being used or if you have manually run the lctl network up command.</para>
</note>
</section>
<section remap="h5">
<title>Description</title>
- <para>The routerstat utility displays LNET router statistics. If no <literal><replaceable>interval</replaceable></literal> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <literal><replaceable>interval</replaceable></literal> (in seconds).</para>
+ <para>The routerstat utility displays LNet router statistics. If no <literal><replaceable>interval</replaceable></literal> is specified, then statistics are sampled and printed only one time. Otherwise, statistics are sampled and printed at the specified <literal><replaceable>interval</replaceable></literal> (in seconds).</para>
</section>
<section remap="h5">
<title>Output</title>
<para> <literal>M</literal></para>
</entry>
<entry>
- <para> Number of messages currently being processed by LNET (The maximum number of messages ever processed by LNET concurrently)</para>
+ <para> Number of messages currently being processed by LNet (The maximum number of messages ever processed by LNet concurrently)</para>
</entry>
</row>
<row>
<para> <literal>E</literal></para>
</entry>
<entry>
- <para> Number of LNET errors</para>
+ <para> Number of LNet errors</para>
</entry>
</row>
<row>
<para> <literal>M</literal></para>
</entry>
<entry>
- <para> Number of messages currently being processed by LNET (The maximum number of messages ever processed by LNET concurrently)</para>
+ <para> Number of messages currently being processed by LNet (The maximum number of messages ever processed by LNet concurrently)</para>
</entry>
</row>
<row>
<para> <literal>E</literal></para>
</entry>
<entry>
- <para> Number of LNET errors per second</para>
+ <para> Number of LNet errors per second</para>
</entry>
</row>
<row>
</indexterm>Lustre Components</title>
<para>An installation of the Lustre software includes a management server
(MGS) and one or more Lustre file systems interconnected with Lustre
- networking (LNET).</para>
+ networking (LNet).</para>
<para>A basic configuration of Lustre file system components is shown in
<xref linkend="understandinglustre.fig.cluster" />.</para>
<figure xml:id="understandinglustre.fig.cluster">
<title>
<indexterm>
<primary>Lustre</primary>
- <secondary>LNET</secondary>
- </indexterm>Lustre Networking (LNET)</title>
- <para>Lustre Networking (LNET) is a custom networking API that provides
+ <secondary>LNet</secondary>
+ </indexterm>Lustre Networking (LNet)</title>
+ <para>Lustre Networking (LNet) is a custom networking API that provides
the communication infrastructure that handles metadata and file I/O data
for the Lustre file system servers and clients. For more information
- about LNET, see
+ about LNet, see
<xref linkend="understandinglustrenetworking" />.</para>
</section>
<section remap="h3">
<?xml version='1.0' encoding='UTF-8'?>
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandinglustrenetworking">
- <title xml:id="understandinglustrenetworking.title">Understanding Lustre Networking (LNET)</title>
- <para>This chapter introduces Lustre networking (LNET). It includes the following sections:</para>
+ <title xml:id="understandinglustrenetworking.title">Understanding Lustre Networking (LNet)</title>
+ <para>This chapter introduces Lustre networking (LNet). It includes the following sections:</para>
<itemizedlist>
<listitem>
<para>
</itemizedlist>
<section xml:id="dbdoclet.50438191_22878">
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
</indexterm><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>understanding</secondary>
- </indexterm> Introducing LNET</title>
+ </indexterm> Introducing LNet</title>
<para>In a cluster using one or more Lustre file systems, the network communication
infrastructure required by the Lustre file system is implemented using the Lustre networking
- (LNET) feature.</para>
- <para>LNET supports many commonly-used network types, such as InfiniBand and IP networks, and
+ (LNet) feature.</para>
+ <para>LNet supports many commonly-used network types, such as InfiniBand and IP networks, and
allows simultaneous availability across multiple network types with routing between them.
Remote direct memory access (RDMA) is permitted when supported by underlying networks using
the appropriate Lustre network driver (LND). High availability and recovery features enable
example <literal>ksocklnd</literal> is the driver which implements the TCP Socket LND that
supports TCP networks. LNDs are loaded into the driver stack, with one LND for each network
type in use.</para>
- <para>For information about configuring LNET, see <xref linkend="configuringlnet"/>.</para>
- <para>For information about administering LNET, see <xref linkend="adminlustrepart3"/>.</para>
+ <para>For information about configuring LNet, see <xref linkend="configuringlnet"/>.</para>
+ <para>For information about administering LNet, see <xref linkend="adminlustrepart3"/>.</para>
</section>
<section xml:id="dbdoclet.50438191_19625">
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>features</secondary>
- </indexterm>Key Features of LNET</title>
- <para>Key features of LNET include:</para>
+ </indexterm>Key Features of LNet</title>
+ <para>Key features of LNet include:</para>
<itemizedlist>
<listitem>
<para>RDMA, when supported by underlying networks</para>
<para>Routing among disparate networks</para>
</listitem>
</itemizedlist>
- <para>LNET permits end-to-end read/write throughput at or near peak bandwidth rates on a variety
+ <para>LNet permits end-to-end read/write throughput at or near peak bandwidth rates on a variety
of network interconnects.</para>
</section>
<section xml:id="idp694976">
<secondary>Networks</secondary>
</indexterm>Lustre Networks</title>
<para>A Lustre network is comprised of clients and servers running the Lustre software. It need
- not be confined to one LNET subnet but can span several networks provided routing is possible
- between the networks. In a similar manner, a single network can have multiple LNET subnets. </para>
- <para>The Lustre networking stack is comprised of two layers, the LNET code module and the LND.
- The LNET layer operates above the LND layer in a manner similar to the way the network layer
- operates above the data link layer. LNET layer is connectionless, asynchronous and does not
+ not be confined to one LNet subnet but can span several networks provided routing is possible
+ between the networks. In a similar manner, a single network can have multiple LNet subnets. </para>
+ <para>The Lustre networking stack is comprised of two layers, the LNet code module and the LND.
+ The LNet layer operates above the LND layer in a manner similar to the way the network layer
+ operates above the data link layer. LNet layer is connectionless, asynchronous and does not
verify that data has been transmitted while the LND layer is connection oriented and typically
does verify data transmission.</para>
- <para>LNETs are uniquely identified by a label comprised of a string corresponding to an LND and
- a number, such as tcp0, o2ib0, or o2ib1, that uniquely identifies each LNET. Each node on an
- LNET has at least one network identifier (NID). A NID is a combination of the address of the
- network interface and the LNET label in the
- form:<literal><replaceable>address</replaceable>@<replaceable>LNET_label</replaceable></literal>.</para>
+ <para>LNets are uniquely identified by a label comprised of a string corresponding to an LND and
+ a number, such as tcp0, o2ib0, or o2ib1, that uniquely identifies each LNet. Each node on an
+ LNet has at least one network identifier (NID). A NID is a combination of the address of the
+ network interface and the LNet label in the
+ form:<literal><replaceable>address</replaceable>@<replaceable>LNet_label</replaceable></literal>.</para>
<para>Examples: <screen>192.168.1.2@tcp0
10.13.24.90@o2ib1</screen></para>
<para>In certain circumstances it might be desirable for Lustre file system traffic to pass
- between multiple LNETs. This is possible using LNET routing. It is important to realize that
- LNET routing is not the same as network routing. For more details about LNET routing, see
+ between multiple LNets. This is possible using LNet routing. It is important to realize that
+ LNet routing is not the same as network routing. For more details about LNet routing, see
<xref xmlns:xlink="http://www.w3.org/1999/xlink" linkend="configuringlnet"/></para>
</section>
<section xml:id="dbdoclet.50438191_20721">
<title><indexterm>
- <primary>LNET</primary>
+ <primary>LNet</primary>
<secondary>supported networks</secondary>
</indexterm>Supported Network Types</title>
- <para>The LNET code module includes LNDs to support many network types including:</para>
+ <para>The LNet code module includes LNDs to support many network types including:</para>
<itemizedlist>
<listitem>
<para> InfiniBand: OpenFabrics OFED (o2ib)</para>
git://git.whamcloud.com / doc / manual.git / commitdiff