X-Git-Url: https://git.whamcloud.com/?a=blobdiff_plain;f=ConfiguringLNet.xml;h=2c450dfaac0b86aa91b1eefaba64a55cd0f03409;hb=1f4d48348d2b5cb0103d7fe7b5c90364e1e2f4a1;hp=da67cbbdfed045e9ba1a1f8e01944a700a2f0ef5;hpb=ef8bb7d5d8377af1d74e2db0e33ed5c9984b3f6f;p=doc%2Fmanual.git diff --git a/ConfiguringLNet.xml b/ConfiguringLNet.xml index da67cbb..2c450df 100755 --- a/ConfiguringLNet.xml +++ b/ConfiguringLNet.xml @@ -1,38 +1,39 @@ Configuring Lustre Networking (LNet) - This chapter describes how to configure Lustre Networking (LNet). It includes the following sections: + This chapter describes how to configure Lustre Networking (LNet). It + includes the following sections: - + - + - + - + - + - + - + - + @@ -45,14 +46,18 @@ required if you are using Infiniband or multiple Ethernet interfaces. The lnetctl utility can be used - to initialize LNet without bringing up any network interfaces. This - gives flexibility to the user to add interfaces after LNet has been - loaded. + to initialize LNet without bringing up any network interfaces. Network + interfaces can be added after configuring LNet via + lnetctl. lnetctl can also be used to + manage an operational LNet. However, if it wasn't initialized by + lnetctl then lnetctl lnet configure + must be invoked before lnetctl can be used to manage + LNet. DLC also introduces a C-API to enable configuring LNet programatically. See -
+
<indexterm> <primary>LNet</primary> <secondary>Configuring LNet</secondary> @@ -82,7 +87,7 @@ </listitem> </itemizedlist> </para> - <section> + <section xml:id="lnet_config.cli_overview"> <title><indexterm> <primary>LNet</primary> <secondary>cli</secondary> @@ -99,7 +104,23 @@ unconfigure LNet.</para> <screen>lnetctl lnet unconfigure</screen> </section> - <section xml:id="dbdoclet.lnetaddshowdelete"> + <section xml:id="lnet_config.show_global_settings"> + <title><indexterm> + <primary>LNet</primary> + <secondary>cli</secondary> + </indexterm>Displaying Global Settings + The active LNet global settings can be displayed using the + lnetctl command shown below: + lnetctl global show + For example: + # lnetctl global show + global: + numa_range: 0 + max_intf: 200 + discovery: 1 + drop_asym_route: 0 +
+
<indexterm><primary>LNet</primary> <secondary>cli</secondary></indexterm>Adding, Deleting and Showing Networks @@ -184,18 +205,20 @@ net: peer_buffer_credits: 0 credits: 256
-
+
<indexterm> <primary>LNet</primary> <secondary>cli</secondary> - </indexterm>Adding, Deleting and Showing Peers + Manual Adding, Deleting and Showing Peers The lnetctl peer add - command is used to add a remote peer to a software - multi-rail configuration. + command is used to manually add a remote peer to a software + multi-rail configuration. For the dynamic peer discovery capability + introduced in Lustre Release 2.11.0, please see + . When configuring peers, use the –-prim_nid option to specify the key or primary nid of the peer node. Then - follow that with the --nid option to specify a set - of comma separated NIDs. + follow that with the --nid option to specify a + set of comma separated NIDs. peer add: add a peer --prim_nid: primary NID of the peer --nid: comma separated list of peer nids (e.g. 10.1.1.2@tcp0) @@ -269,15 +292,75 @@ net: --nid: comma separated list of peer nids (e.g. 10.1.1.2@tcp0) prim_nid should always be specified. The prim_nid identifies the peer. If the - prim_nid is the only one specified, then the entire - peer is deleted. + prim_nid is the only one specified, then the + entire peer is deleted. Example of deleting a single nid of a peer (10.10.10.3@tcp): lnetctl peer del --prim_nid 10.10.10.2@tcp --nid 10.10.10.3@tcp Example of deleting the entire peer: lnetctl peer del --prim_nid 10.10.10.2@tcp -
+
+ <indexterm> + <primary>LNet</primary> + <secondary>cli</secondary> + <tertiary>dynamic discovery</tertiary> + </indexterm>Dynamic Peer Discovery +
+ Overview + Dynamic Discovery (DD) is a feature that allows nodes to + dynamically discover a peer's interfaces without having to explicitly + configure them. This is very useful for Multi-Rail (MR) + configurations. In large clusters, there could be hundreds of nodes + and having to configure MR peers on each node becomes error prone. + Dynamic Discovery is enabled by default and uses a new protocol based + on LNet pings to discover the interfaces of the remote peers on first + message. +
+
+ Protocol + When LNet on a node is requested to send a message to a peer it + first attempts to ping the peer. The reply to the ping contains the + peer's NIDs as well as a feature bit outlining what the peer supports. + Dynamic Discovery adds a Multi-Rail feature bit. If the peer is + Multi-Rail capable, it sets the MR bit in the ping reply. When the + node receives the reply it checks the MR bit, and if it is set it then + pushes its own list of NIDs to the peer using a new PUT message, + referred to as a "push ping". After this brief protocol, both the peer + and the node will have each other's list of interfaces. The MR + algorithm can then proceed to use the list of interfaces of the + corresponding peer. + If the peer is not MR capable, it will not set the MR feature + bit in the ping reply. The node will understand that the peer is + not MR capable and will only use the interface provided by upper + layers for sending messages. +
+
+ Dynamic Discovery and User-space Configuration + It is possible to configure the peer manually while Dynamic + Discovery is running. Manual peer configuration always takes precedence + over Dynamic Discovery. If there is a discrepancy between the manual + configuration and the dynamically discovered information, a warning is + printed. +
+
+ Configuration + Dynamic Discovery is very light on the configuration side. It can + only be turned on or turned off. To turn the feature on or off, the + following command is used: + lnetctl set discovery [0 | 1] + To check the current discovery setting, the + lnetctl global show command can be used as shown in + . +
+
+ Initiating Dynamic Discovery on Demand + It is possible to initiate the Dynamic Discovery protocol on demand + without having to wait for a message to be sent to the peer. This can + be done with the following command: + lnetctl discover <peer_nid> [<peer_nid> ...] +
+
<indexterm> <primary>LNet</primary> @@ -297,7 +380,8 @@ net: Example: lnetctl route add --net tcp2 --gateway 192.168.205.130@tcp1 --hop 2 --prio 1</screen> - <para>Routes can be deleted via the following <literal>lnetctl</literal> command.</para> + <para>Routes can be deleted via the following <literal>lnetctl</literal> + command.</para> <screen>lnetctl route del: delete a route --net: net name (ex tcp0) --gateway: gateway nid (ex 10.1.1.2@tcp) @@ -425,6 +509,37 @@ set large_buffers: set large routing buffers > lnetctl set small_buffers 0 > lnetctl set large_buffers 0</screen> </section> + <section condition='l2D' xml:id="lnet_config.asym_route"> + <title><indexterm> + <primary>LNet</primary> + <secondary>cli</secondary> + <tertiary>asymmetrical route</tertiary> + </indexterm>Asymmetrical Routes +
+ Overview + An asymmetrical route is when a message from a remote peer is + coming through a router that is not known by this node + to reach the remote peer. + Asymmetrical routes can be an issue when debugging network, and + allowing them also opens the door to attacks where hostile clients + inject data to the servers. + So it is possible to activate a check in LNet, that will detect + any asymmetrical route message and drop it. +
+
+ Configuration + In order to switch asymmetric route detection on or off, the + following command is used: + lnetctl set drop_asym_route [0 | 1] + This command works on a per-node basis. This means each node in a + Lustre cluster can decide whether it accepts asymmetrical route + messages. + To check the current drop_asym_route setting, the + lnetctl global show command can be used as shown in + . + By default, asymmetric route detection is off. +
+
<indexterm> <primary>LNet</primary> @@ -515,13 +630,15 @@ lnetctl export > FILE.yaml</screen> integer represents the CPT to associate the network interface with> seq_no: <integer. Optional. User generated, and is passed back in the YAML error block></screen> - <para>Both seq_no and detail fields do not appear in the show output.</para> + <para>Both seq_no and detail fields do not appear in the show output. + </para> </section> <section> <title><indexterm> <primary>LNet</primary> <secondary>buffer yaml syntax</secondary> - </indexterm>Enable Routing and Adjust Router Buffer Configuration + Enable Routing and Adjust Router Buffer Configuration + routing: - tiny: <Integer. Tiny buffers> @@ -553,14 +670,14 @@ lnetctl export > FILE.yaml hop: <an integer between 1 and 255. Optional> detail: <This is only applicable for show commands. 1 - output detailed info. 0. basic output> seq_no: <integer. Optional. User generated, and is passed back in the YAML error block> - Both seq_no and detail fields do not appear in the show output. + Both seq_no and detail fields do not appear in the show output. +
-
- <indexterm><primary>LNet</primary></indexterm> - - Overview of LNet Module Parameters +
+ <indexterm><primary>LNet</primary></indexterm> + Overview of LNet Module Parameters LNet kernel module (lnet) parameters specify how LNet is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with @@ -581,7 +698,8 @@ lnetctl export > FILE.yaml be used at a time): - networks - Specifies the networks to be used. + networks - Specifies the networks to be used. + ip2nets - Lists globally-available @@ -590,18 +708,20 @@ lnetctl export > FILE.yaml lookup. - See and for more details. + See and + for more details. To set up routing between networks, use: - routes - Lists networks and the NIDs of routers that forward to them. + routes - Lists networks and the NIDs of + routers that forward to them. - See for more details. + See for more details. A router checker can be configured to enable Lustre nodes to detect router health status, avoid routers that appear dead, and reuse those that restore service after failures. See for more details. + linkend="lnet_router_checker"/> for more details. For a complete reference to the LNet module parameters, see LNet Options. @@ -631,12 +751,15 @@ lnetctl export > FILE.yaml the mount command, use the lctl command. To display MDS NIDs, run on the MDS : lctl list_nids - To determine if a client can reach the MDS using a particular NID, run on the client: + To determine if a client can reach the MDS using a particular NID, + run on the client: lctl which_nid MDS_NID
-
- <indexterm><primary>LNet</primary><secondary>module parameters</secondary></indexterm>Setting the LNet Module networks Parameter +
+ <indexterm><primary>LNet</primary> + <secondary>module parameters</secondary> + </indexterm>Setting the LNet Module networks Parameter If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the lustre.conf file @@ -647,7 +770,8 @@ lnetctl export > FILE.yaml This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface: options lnet networks=tcp0(eth0),o2ib(ib0) - This example specifies that the Lustre node will use the TCP/IP interface eth1: + This example specifies that the Lustre node will use the TCP/IP + interface eth1: options lnet networks=tcp0(eth1) Depending on the network design, it may be necessary to specify explicit interfaces. To explicitly specify that interface @@ -667,7 +791,9 @@ lnetctl export > FILE.yaml not used for routing decisions.
- <indexterm><primary>configuring</primary><secondary>multihome</secondary></indexterm>Multihome Server Example + <indexterm><primary>configuring</primary> + <secondary>multihome</secondary></indexterm>Multihome Server Example + If a server with multiple IP addresses (multihome server) is connected to a Lustre network, certain configuration setting are required. An example illustrating these setting consists of a @@ -692,7 +818,8 @@ lnetctl export > FILE.yaml interface and a TCP/IP interface for administration. - To set the networks option for this example: + To set the networks option for this example: + On each server, svr1 and @@ -731,8 +858,10 @@ lnetctl export > FILE.yaml
-
- <indexterm><primary>LNet</primary><secondary>ip2nets</secondary></indexterm>Setting the LNet Module ip2nets Parameter +
+ <indexterm><primary>LNet</primary> + <secondary>ip2nets</secondary> + </indexterm>Setting the LNet Module ip2nets Parameter The ip2nets option is typically used when a single, universal lustre.conf file is run on all servers and clients. Each node identifies the locally available @@ -772,7 +901,8 @@ lnetctl export > FILE.yaml lustre.conf file on each server and client: options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \ tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"' - Each entry in ip2nets is referred to as a 'rule'. + Each entry in ip2nets is referred to as a + 'rule'. The order of LNet entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in lustre.conf will be @@ -789,8 +919,8 @@ tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"' network. Multi-rail deprecates the kernel parsing of ip2nets. ip2nets - patterns are matched in user space and translated into Network interfaces - to be added into the system. + patterns are matched in user space and translated into Network + interfaces to be added into the system. The first interface that matches the IP pattern will be used when adding a network interface. If an interface is explicitly specified as well as a pattern, the @@ -798,8 +928,9 @@ tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"' explicitly-defined interface. For example, tcp(eth0) 192.168.*.3 and there exists in the system eth0 == 192.158.19.3 and - eth1 == 192.168.3.3, then the configuration will fail, - because the pattern contradicts the interface specified. + eth1 == 192.168.3.3, then the configuration will + fail, because the pattern contradicts the interface specified. + A clear warning will be displayed if inconsistent configuration is encountered. You could use the following command to configure ip2nets: @@ -820,9 +951,10 @@ tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"' 0: 192.168.*.*
-
- <indexterm><primary>LNet</primary><secondary>routes</secondary></indexterm>Setting - the LNet Module routes Parameter +
+ <indexterm><primary>LNet</primary> + <secondary>routes</secondary></indexterm>Setting the LNet Module routes + Parameter The LNet module routes parameter is used to identify routers in a Lustre configuration. These parameters are set in modprobe.conf on each Lustre node. @@ -860,17 +992,19 @@ tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"' lctl network configure
-
- <indexterm><primary>LNet</primary><secondary>testing</secondary></indexterm>Testing - the LNet Configuration +
+ <indexterm><primary>LNet</primary> + <secondary>testing</secondary></indexterm>Testing the LNet + Configuration After configuring Lustre Networking, it is highly recommended that you test your LNet configuration using the LNet Self-Test provided with the Lustre software. For more information about using LNet Self-Test, see .
-
- <indexterm><primary>LNet</primary><secondary>route - checker</secondary></indexterm>Configuring the Router Checker +
+ <indexterm><primary>LNet</primary> + <secondary>route checker</secondary> + </indexterm>Configuring the Router Checker In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the @@ -928,7 +1062,8 @@ lctl network configure options lnet check_routers_before_use=on - The router checker obtains the following information from each router: + The router checker obtains the following information from each router: + Time the router was disabled @@ -945,17 +1080,17 @@ lctl network configure If 100 packets have been sent successfully through a router, the sent-packets counter for that router will have a value of 100.
-
- <indexterm><primary>LNet</primary><secondary>best - practice</secondary></indexterm>Best Practices for LNet - Options +
+ <indexterm><primary>LNet</primary> + <secondary>best practice</secondary> + </indexterm>Best Practices for LNet Options For the networks, ip2nets, and routes options, follow these best practices to avoid configuration errors. -
- <indexterm><primary>LNet</primary><secondary>escaping commas - with quotes</secondary></indexterm>Escaping commas with - quotes +
+ <indexterm><primary>LNet</primary> + <secondary>escaping commas with quotes</secondary> + </indexterm>Escaping commas with quotes Depending on the Linux distribution, commas may need to be escaped using single or double quotes. In the extreme case, the options entry would look like this: @@ -969,9 +1104,9 @@ lctl network configure NID' message generally points to an error in the LNet module configuration.
-
- <indexterm><primary>LNet</primary><secondary>comments</secondary></indexterm>Including - comments +
+ <indexterm><primary>LNet</primary> + <secondary>comments</secondary></indexterm>Including comments Place the semicolon terminating a comment immediately after the comment. LNet silently ignores everything between the # character at the