1 <?xml version='1.0' encoding='UTF-8'?>
2 <!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configuringlnet">
3 <title xml:id="configuringlnet.title">Configuring Lustre Networking (LNET)</title>
4 <para>This chapter describes how to configure Lustre Networking (LNET). It includes the following sections:</para>
7 <para><xref linkend="dbdoclet.50438216_15201"/>
11 <para><xref linkend="dbdoclet.50438216_33148"/>
15 <para><xref linkend="dbdoclet.50438216_46279"/>
19 <para><xref linkend="dbdoclet.50438216_31414"/>
23 <para><xref linkend="dbdoclet.50438216_71227"/>
27 <para><xref linkend="dbdoclet.50438216_10523"/>
31 <para><xref linkend="dbdoclet.50438216_35668"/>
35 <para><xref linkend="dbdoclet.50438216_15200"/>
40 <para>Configuring LNET is optional.</para>
41 <para> LNET will use the first TCP/IP interface it discovers on a
42 system (<literal>eth0</literal>) if it's loaded using the
43 <literal>lctl network up</literal>. If this network configuration is
44 sufficient, you do not need to configure LNET. LNet configuration is
45 required if you are using Infiniband or multiple Ethernet
47 <para condition='l27'>The <literal>lnetctl</literal> utility can be used
48 to initialize LNET without bringing up any network interfaces. This
49 gives flexibility to the user to add interfaces after LNET has been
51 <para condition='l27'>DLC also introduces a C-API to enable
52 configuring LNET programatically. See <xref
53 linkend="lnetconfigurationapi"/></para>
55 <section xml:id="dbdoclet.50438216_15201" condition='l27'>
57 <primary>LNET</primary>
58 <secondary>Configuring LNET</secondary>
59 </indexterm>Configuring LNET via <literal>lnetctl</literal></title>
60 <para>The <literal>lnetctl</literal> utility can be used to initialize
61 and configure the LNET kernel module after it has been loaded via
62 <literal>modprobe</literal>. In general the lnetctl format is as
64 <screen>lnetctl cmd subcmd [options]</screen>
65 <para>The following configuration items are managed by the tool:</para>
69 <para>Configuring/unconfiguring LNET</para>
72 <para>Adding/removing/showing Networks</para>
75 <para>Adding/removing/showing Routes</para>
78 <para>Enabling/Disabling routing</para>
81 <para>Configuring Router Buffer Pools</para>
87 <primary>LNET</primary>
88 <secondary>cli</secondary>
89 </indexterm>Configuring LNET</title>
90 <para>After LNET has been loaded via <literal>modprobe</literal>,
91 <literal>lnetctl</literal> utility can be used to configure LNET
92 without bringing up networks which are specified in the module
93 parameters. It can also be used to configure network interfaces
94 specified in the module prameters by providing the
95 <literal>--all</literal> option.</para>
96 <screen>lnetctl lnet configure [--all]
97 # --all: load NI configuration from module parameters</screen>
98 <para>The <literal>lnetctl</literal> utility can also be used to
99 unconfigure LNET.</para>
100 <screen>lnetctl lnet unconfigure</screen>
104 <primary>LNET</primary>
105 <secondary>cli</secondary>
106 </indexterm>Adding, Deleting and Showing networks</title>
107 <para>Networks can be added and deleted after the LNET kernel module
109 <screen>lnetctl net add: add a network
110 --net: net name (ex tcp0)
111 --if: physical interface (ex eth0)
112 --peer_timeout: time to wait before declaring a peer dead
113 --peer_credits: defines the max number of inflight messages
114 --peer_buffer_credits: the number of buffer credits per peer
115 --credits: Network Interface credits
116 --cpts: CPU Partitions configured net uses
117 --help: display this help text
120 lnetctl net add --net tcp2 --if eth0
121 --peer_timeout 180 --peer_credits 8</screen>
122 <para>Networks can be deleted as shown below:</para>
123 <screen>net del: delete a network
124 --net: net name (ex tcp0)
127 lnetctl net del --net tcp2</screen>
128 <para>All or a subset of the configured networks can be shown. The
129 output can be non-verbose
131 <screen>net show: show networks
132 --net: net name (ex tcp0) to filter on
133 --verbose: display detailed output per network
137 lnetctl net show --verbose
138 lnetctl net show --net tcp2 --verbose</screen>
139 <para>Below are examples of non-detailed and detailed network
140 configuration show.</para>
141 <screen># non-detailed show
142 > lnetctl net show --net tcp2
144 - nid: 192.168.205.130@tcp2
150 > lnetctl net show --net tcp2 --verbose
152 - nid: 192.168.205.130@tcp2
159 peer_buffer_credits: 0
160 credits: 256</screen>
164 <primary>LNET</primary>
165 <secondary>cli</secondary>
166 </indexterm>Adding, Deleting and Showing routes</title>
167 <para>A set of routes can be added to identify how LNET messages are
169 <screen>lnetctl route add: add a route
170 --net: net name (ex tcp0) LNET message is destined to.
171 The can not be a local network.
172 --gateway: gateway node nid (ex 10.1.1.2@tcp) to route
173 all LNET messaged destined for the identified
175 --hop: number of hops to final destination
176 (1 < hops < 255)
177 --priority: priority of route (0 - highest prio)
180 lnetctl route add --net tcp2 --gateway 192.168.205.130@tcp1 --hop 2 --prio 1</screen>
181 <para>Routes can be deleted via the following <literal>lnetctl</literal> command.</para>
182 <screen>lnetctl route del: delete a route
183 --net: net name (ex tcp0)
184 --gateway: gateway nid (ex 10.1.1.2@tcp)
187 lnetctl route del --net tcp2 --gateway 192.168.205.130@tcp1</screen>
188 <para>Configured routes can be shown via the following
189 <literal>lnetctl</literal> command.</para>
190 <screen>lnetctl route show: show routes
191 --net: net name (ex tcp0) to filter on
192 --gateway: gateway nid (ex 10.1.1.2@tcp) to filter on
193 --hop: number of hops to final destination
194 (1 < hops < 255) to filter on
195 --priority: priority of route (0 - highest prio)
197 --verbose: display detailed output per route
204 lnetctl route show --verbose</screen>
205 <para>When showing routes the <literal>--verbose</literal> option
206 outputs more detailed information. All show and error output are in
207 YAML format. Below are examples of both non-detailed and detailed
208 route show output.</para>
209 <screen>#Non-detailed output
213 gateway: 192.168.205.130@tcp1
216 > lnetctl route show --verbose
219 gateway: 192.168.205.130@tcp1
226 <primary>LNET</primary>
227 <secondary>cli</secondary>
228 </indexterm>Enabling and Disabling Routing</title>
229 <para>When an LNET node is configured as a router it will route LNet
230 messages not destined to itself. This feature can be enabled or
231 disabled as follows.</para>
232 <screen>lnetctl set routing [0 | 1]
233 # 0 - disable routing feature
234 # 1 - enable routing feature</screen>
238 <primary>LNET</primary>
239 <secondary>cli</secondary>
240 </indexterm>Showing routing information</title>
241 <para>When routing is enabled on a node, the tiny, small and large
242 routing buffers are allocated. See <xref
243 linkend="dbdoclet.50438272_73839"/> for more details on router
244 buffers. This information can be shown as follows:</para>
245 <screen>lnetctl routing show: show routing information
248 lnetctl routing show</screen>
249 <para>An example of the show output:</para>
250 <screen>> lnetctl routing show
272 <primary>LNET</primary>
273 <secondary>cli</secondary>
274 </indexterm>Configuring Routing Buffers</title>
275 <para> The routing buffers values configured specify the number of
276 buffers in each of the tiny, small and large groups.</para>
277 <para>It is often desirable to configure the tiny, small and large
278 routing buffers to some values other than the default. These values
279 are global values, when set they are used by all configured CPU
280 partitions. If routing is enabled then the values set take effect
281 immediately. If a larger number of buffers is specified, then
282 buffers are allocated to satisfy the configuration change. If fewer
283 buffers are configured then the excess buffers are freed as they
284 become unused. If routing is not set the values are not changed.
285 The buffer values are reset to default if routing is turned off and
287 <para>The <literal>lnetctl</literal> 'set' command can be
288 used to set these buffer values. A VALUE greater than 0
289 will set the number of buffers accordingly. A VALUE of 0
290 will reset the number of buffers to system defaults.</para>
291 <screen>set tiny_buffers:
292 set tiny routing buffers
293 VALUE must be greater than or equal to 0
295 set small_buffers: set small routing buffers
296 VALUE must be greater than or equal to 0
298 set large_buffers: set large routing buffers
299 VALUE must be greater than or equal to 0</screen>
300 <para>Usage examples:</para>
301 <screen>> lnetctl set tiny_buffers 4096
302 > lnetctl set small_buffers 8192
303 > lnetctl set large_buffers 2048</screen>
304 <para>The buffers can be set back to the default values as follows:</para>
305 <screen>> lnetctl set tiny_buffers 0
306 > lnetctl set small_buffers 0
307 > lnetctl set large_buffers 0</screen>
311 <primary>LNET</primary>
312 <secondary>cli</secondary>
313 </indexterm>Importing YAML Configuration File</title>
314 <para>Configuration can be described in YAML format and can be fed
315 into the <literal>lnetctl</literal> utility. The
316 <literal>lnetctl</literal> utility parses the YAML file and performs
317 the specified operation on all entities described there in. If no
318 operation is defined in the command as shown below, the default
319 operation is 'add'. The YAML syntax is described in a later
320 section.</para> <screen>lnetctl import FILE.yaml
321 lnetctl import < FILE.yaml</screen>
322 <para>The '<literal>lnetctl</literal> import' command provides three
323 optional parameters to define the operation to be performed on the
324 configuration items described in the YAML file.</para>
325 <screen># if no options are given to the command the "add" command is assumed
327 lnetctl import --add FILE.yaml
328 lnetctl import --add < FILE.yaml
330 # to delete all items described in the YAML file
331 lnetctl import --del FILE.yaml
332 lnetctl import --del < FILE.yaml
334 # to show all items described in the YAML file
335 lnetctl import --show FILE.yaml
336 lnetctl import --show < FILE.yaml</screen>
340 <primary>LNET</primary>
341 <secondary>cli</secondary>
342 </indexterm>Exporting Configuration in YAML format</title>
343 <para><literal>lnetctl</literal> utility provides the 'export'
344 command to dump current LNET configuration in YAML format </para>
345 <screen>lnetctl export FILE.yaml
346 lnetctl export > FILE.yaml</screen>
350 <primary>LNET</primary>
351 <secondary>cli</secondary>
352 </indexterm>Showing LNET Traffic Statistics</title>
353 <para><literal>lnetctl</literal> utility can dump the LNET traffic
354 statistiscs as follows</para>
355 <screen>lnetctl stats show</screen>
359 <primary>LNET</primary>
360 <secondary>yaml syntax</secondary>
361 </indexterm>YAML Syntax</title>
362 <para>The <literal>lnetctl</literal> utility can take in a YAML file
363 describing the configuration items that need to be operated on and
364 perform one of the following operations: add, delete or show on the
365 items described there in.</para>
366 <para>Net, routing and route YAML blocks are all defined as a YAML
367 sequence, as shown in the following sections. The stats YAML block
368 is a YAML object. Each sequence item can take a seq_no field. This
369 seq_no field is returned in the error block. This allows the caller
370 to associate the error with the item that caused the error. The
371 <literal>lnetctl</literal> utilty does a best effort at configuring
372 items defined in the YAML file. It does not stop processing the file
373 at the first error.</para>
374 <para>Below is the YAML syntax describing the various
375 configuration elements which can be operated on via DLC. Not all
376 YAML elements are requied for all operations (add/delete/show).
377 The system ignores elements which are not pertinent to the requested
381 <primary>LNET</primary>
382 <secondary>network yaml syntax</secondary>
383 </indexterm>Network Configuration</title>
386 - net: <network. Ex: tcp or o2ib>
388 0: <physical interface>
389 detail: <This is only applicable for show command. 1 - output detailed info. 0 - basic output>
391 peer_timeout: <Integer. Timeout before consider a peer dead>
392 peer_credits: <Integer. Transmit credits for a peer>
393 peer_buffer_credits: <Integer. Credits available for receiving messages>
394 credits: <Integer. Network Interface credits>
395 SMP: <An array of integers of the form: "[x,y,...]", where each
396 integer represents the CPT to associate the network interface
397 with> seq_no: <integer. Optional. User generated, and is
398 passed back in the YAML error block></screen>
399 <para>Both seq_no and detail fields do not appear in the show output.</para>
403 <primary>LNET</primary>
404 <secondary>buffer yaml syntax</secondary>
405 </indexterm>Enable Routing and Adjust Router Buffer Configuration</title>
408 - tiny: <Integer. Tiny buffers>
409 small: <Integer. Small buffers>
410 large: <Integer. Large buffers>
411 enable: <0 - disable routing. 1 - enable routing>
412 seq_no: <Integer. Optional. User generated, and is passed back in the YAML error block></screen>
413 <para>The seq_no field does not appear in the show output</para>
417 <primary>LNET</primary>
418 <secondary>statistics yaml syntax</secondary>
419 </indexterm>Show Statistics</title>
422 seq_no: <Integer. Optional. User generated, and is passed back in the YAML error block></screen>
423 <para>The seq_no field does not appear in the show output</para>
427 <primary>LNET</primary>
428 <secondary>router yaml syntax</secondary>
429 </indexterm>Route Configuration</title>
432 - net: <network. Ex: tcp or o2ib>
433 gateway: <nid of the gateway in the form <ip>@<net>: Ex: 192.168.29.1@tcp>
434 hop: <an integer between 1 and 255. Optional>
435 detail: <This is only applicable for show commands. 1 - output detailed info. 0. basic output>
436 seq_no: <integer. Optional. User generated, and is passed back in the YAML error block></screen>
437 <para>Both seq_no and detail fields do not appear in the show output.</para>
441 <section xml:id="dbdoclet.50438216_33148">
442 <title><indexterm><primary>LNET</primary></indexterm>
444 Overview of LNET Module Parameters</title>
445 <para>LNET kernel module (lnet) parameters specify how LNet is to be
446 configured to work with Lustre, including which NICs will be
447 configured to work with Lustre and the routing to be used with
449 <para>Parameters for LNET can be specified in the
450 <literal>/etc/modprobe.d/lustre.conf</literal> file. In some cases
451 the parameters may have been stored in
452 <literal>/etc/modprobe.conf</literal>, but this has been deprecated
453 since before RHEL5 and SLES10, and having a separate
454 <literal>/etc/modprobe.d/lustre.conf</literal> file simplifies
455 administration and distribution of the Lustre networking
456 configuration. This file contains one or more entries with the
458 <screen>options lnet <replaceable>parameter</replaceable>=<replaceable>value</replaceable></screen>
459 <para>To specify the network interfaces that are to be used for
460 Lustre, set either the <literal>networks</literal> parameter or the
461 <literal>ip2nets</literal> parameter (only one of these parameters can
462 be used at a time):</para>
465 <para><literal>networks</literal> - Specifies the networks to be used.</para>
468 <para><literal>ip2nets</literal> - Lists globally-available
469 networks, each with a range of IP addresses. LNET then identifies
470 locally-available networks through address list-matching
474 <para>See <xref linkend="dbdoclet.50438216_46279"/> and <xref linkend="dbdoclet.50438216_31414"/> for more details.</para>
475 <para>To set up routing between networks, use:</para>
478 <para><literal>routes</literal> - Lists networks and the NIDs of routers that forward to them.</para>
481 <para>See <xref linkend="dbdoclet.50438216_71227"/> for more details.</para>
482 <para>A <literal>router</literal> checker can be configured to enable
483 Lustre nodes to detect router health status, avoid routers that appear
484 dead, and reuse those that restore service after failures. See <xref
485 linkend="dbdoclet.50438216_35668"/> for more details.</para>
486 <para>For a complete reference to the LNET module parameters, see
487 <emphasis><xref linkend="configurationfilesmoduleparameters"/>LNET
488 Options</emphasis>.</para>
490 <para>We recommend that you use 'dotted-quad' notation for
491 IP addresses rather than host names to make it easier to read debug
492 logs and debug configurations with multiple interfaces.</para>
495 <title><indexterm><primary>LNET</primary><secondary>using
496 NID</secondary></indexterm>Using a Lustre Network Identifier (NID)
497 to Identify a Node</title>
498 <para>A Lustre network identifier (NID) is used to uniquely identify
499 a Lustre network endpoint by node ID and network type. The format of
501 <screen><replaceable>network_id</replaceable>@<replaceable>network_type</replaceable></screen>
502 <para>Examples are:</para>
503 <screen>10.67.73.200@tcp0
504 10.67.75.100@o2ib</screen>
505 <para>The first entry above identifies a TCP/IP node, while the
506 second entry identifies an InfiniBand node.</para>
507 <para>When a mount command is run on a client, the client uses the
508 NID of the MDS to retrieve configuration information. If an MDS has
509 more than one NID, the client should use the appropriate NID for its
510 local network.</para>
511 <para>To determine the appropriate NID to specify in
512 the mount command, use the <literal>lctl</literal> command. To
513 display MDS NIDs, run on the MDS :</para>
514 <screen>lctl list_nids</screen>
515 <para>To determine if a client can reach the MDS using a particular NID, run on the client:</para>
516 <screen>lctl which_nid <replaceable>MDS_NID</replaceable></screen>
519 <section xml:id="dbdoclet.50438216_46279">
520 <title><indexterm><primary>LNET</primary><secondary>module parameters</secondary></indexterm>Setting the LNet Module networks Parameter</title>
521 <para>If a node has more than one network interface, you'll
522 typically want to dedicate a specific interface to Lustre. You can do
523 this by including an entry in the <literal>lustre.conf</literal> file
524 on the node that sets the LNET module <literal>networks</literal>
526 <screen>options lnet networks=<replaceable>comma-separated list of
527 networks</replaceable></screen>
528 <para>This example specifies that a Lustre node will use a TCP/IP
529 interface and an InfiniBand interface:</para>
530 <screen>options lnet networks=tcp0(eth0),o2ib(ib0)</screen>
531 <para>This example specifies that the Lustre node will use the TCP/IP interface <literal>eth1</literal>:</para>
532 <screen>options lnet networks=tcp0(eth1)</screen>
533 <para>Depending on the network design, it may be necessary to specify
534 explicit interfaces. To explicitly specify that interface
535 <literal>eth2</literal> be used for network <literal>tcp0</literal>
536 and <literal>eth3</literal> be used for <literal>tcp1</literal> , use
538 <screen>options lnet networks=tcp0(eth2),tcp1(eth3)</screen>
539 <para>When more than one interface is available during the network
540 setup, Lustre chooses the best route based on the hop count. Once the
541 network connection is established, Lustre expects the network to stay
542 connected. In a Lustre network, connections do not fail over to
543 another interface, even if multiple interfaces are available on the
546 <para>LNET lines in <literal>lustre.conf</literal> are only used by
547 the local node to determine what to call its interfaces. They are
548 not used for routing decisions.</para>
551 <title><indexterm><primary>configuring</primary><secondary>multihome</secondary></indexterm>Multihome Server Example</title>
552 <para>If a server with multiple IP addresses (multihome server) is
553 connected to a Lustre network, certain configuration setting are
554 required. An example illustrating these setting consists of a
555 network with the following nodes:</para>
558 <para> Server svr1 with three TCP NICs (<literal>eth0</literal>,
559 <literal>eth1</literal>, and <literal>eth2</literal>) and an
560 InfiniBand NIC.</para>
563 <para> Server svr2 with three TCP NICs (<literal>eth0</literal>,
564 <literal>eth1</literal>, and <literal>eth2</literal>) and an
565 InfiniBand NIC. Interface eth2 will not be used for Lustre
569 <para> TCP clients, each with a single TCP interface.</para>
572 <para> InfiniBand clients, each with a single Infiniband
573 interface and a TCP/IP interface for administration.</para>
576 <para>To set the <literal>networks</literal> option for this example:</para>
579 <para> On each server, <literal>svr1</literal> and
580 <literal>svr2</literal>, include the following line in the
581 <literal>lustre.conf</literal> file:</para>
584 <screen>options lnet networks=tcp0(eth0),tcp1(eth1),o2ib</screen>
587 <para> For TCP-only clients, the first available non-loopback IP
588 interface is used for <literal>tcp0</literal>. Thus, TCP clients
589 with only one interface do not need to have options defined in
590 the <literal>lustre.conf</literal> file.</para>
593 <para> On the InfiniBand clients, include the following line in
594 the <literal>lustre.conf</literal> file:</para>
597 <screen>options lnet networks=o2ib</screen>
599 <para>By default, Lustre ignores the loopback
600 (<literal>lo0</literal>) interface. Lustre does not ignore IP
601 addresses aliased to the loopback. If you alias IP addresses to
602 the loopback interface, you must specify all Lustre networks using
603 the LNET networks parameter.</para>
606 <para>If the server has multiple interfaces on the same subnet,
607 the Linux kernel will send all traffic using the first configured
608 interface. This is a limitation of Linux, not Lustre. In this
609 case, network interface bonding should be used. For more
610 information about network interface bonding, see <xref
611 linkend="settingupbonding"/>.</para>
615 <section xml:id="dbdoclet.50438216_31414">
616 <title><indexterm><primary>LNET</primary><secondary>ip2nets</secondary></indexterm>Setting the LNet Module ip2nets Parameter</title>
617 <para>The <literal>ip2nets</literal> option is typically used when a
618 single, universal <literal>lustre.conf</literal> file is run on all
619 servers and clients. Each node identifies the locally available
620 networks based on the listed IP address patterns that match the
621 node's local IP addresses.</para>
622 <para>Note that the IP address patterns listed in the
623 <literal>ip2nets</literal> option are <emphasis>only</emphasis> used
624 to identify the networks that an individual node should instantiate.
625 They are <emphasis>not</emphasis> used by LNET for any other
626 communications purpose.</para>
627 <para>For the example below, the nodes in the network have these IP
631 <para> Server svr1: <literal>eth0</literal> IP address
632 <literal>192.168.0.2</literal>, IP over Infiniband
633 (<literal>o2ib</literal>) address
634 <literal>132.6.1.2</literal>.</para>
637 <para> Server svr2: <literal>eth0</literal> IP address
638 <literal>192.168.0.4</literal>, IP over Infiniband
639 (<literal>o2ib</literal>) address
640 <literal>132.6.1.4</literal>.</para>
643 <para> TCP clients have IP addresses
644 <literal>192.168.0.5-255.</literal></para>
647 <para> Infiniband clients have IP over Infiniband
648 (<literal>o2ib</literal>) addresses <literal>132.6.[2-3].2, .4,
649 .6, .8</literal>.</para>
652 <para>The following entry is placed in the
653 <literal>lustre.conf</literal> file on each server and client:</para>
654 <screen>options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \
655 tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"'</screen>
656 <para>Each entry in <literal>ip2nets</literal> is referred to as a 'rule'.</para>
657 <para>The order of LNET entries is important when configuring servers.
658 If a server node can be reached using more than one network, the first
659 network specified in <literal>lustre.conf</literal> will be
661 <para>Because <literal>svr1</literal> and <literal>svr2</literal>
662 match the first rule, LNET uses <literal>eth0</literal> for
663 <literal>tcp0</literal> on those machines. (Although
664 <literal>svr1</literal> and <literal>svr2</literal> also match the
665 second rule, the first matching rule for a particular network is
667 <para>The <literal>[2-8/2]</literal> format indicates a range of 2-8
668 stepped by 2; that is 2,4,6,8. Thus, the clients at
669 <literal>132.6.3.5</literal> will not find a matching o2ib
672 <section xml:id="dbdoclet.50438216_71227">
673 <title><indexterm><primary>LNET</primary><secondary>routes</secondary></indexterm>Setting
674 the LNet Module routes Parameter</title>
675 <para>The LNET module routes parameter is used to identify routers in
676 a Lustre configuration. These parameters are set in
677 <literal>modprobe.conf</literal> on each Lustre node. </para>
678 <para>Routes are typicall set to connect to segregated subnetworks
679 or to cross connect two different types of networks such as tcp and
681 <para>The LNET routes parameter specifies a colon-separated list of
682 router definitions. Each route is defined as a network number,
683 followed by a list of routers:</para>
684 <screen>routes=<replaceable>net_type router_NID(s)</replaceable></screen>
685 <para>This example specifies bi-directional routing in which TCP
686 clients can reach Lustre resources on the IB networks and IB servers
687 can access the TCP networks:</para>
688 <screen>options lnet 'ip2nets="tcp0 192.168.0.*; \
689 o2ib0(ib0) 132.6.1.[1-128]"' 'routes="tcp0 132.6.1.[1-8]@o2ib0; \
690 o2ib0 192.16.8.0.[1-8]@tcp0"'</screen>
691 <para>All LNET routers that bridge two networks are equivalent. They
692 are not configured as primary or secondary, and the load is balanced
693 across all available routers.</para>
694 <para>The number of LNET routers is not limited. Enough routers should
695 be used to handle the required file serving bandwidth plus a 25
696 percent margin for headroom.</para>
698 <title><indexterm><primary>LNET</primary><secondary>routing
699 example</secondary></indexterm>Routing Example</title>
700 <para>On the clients, place the following entry in the
701 <literal>lustre.conf</literal> file</para>
702 <screen>lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0"</screen>
703 <para>On the router nodes, use:</para>
704 <screen>lnet networks="tcp o2ib" forwarding=enabled </screen>
705 <para>On the MDS, use the reverse as shown below:</para>
706 <screen>lnet networks="o2ib0" routes="tcp0 132.6.1.[1-8]@o2ib0" </screen>
707 <para>To start the routers, run:</para>
708 <screen>modprobe lnet
709 lctl network configure</screen>
712 <section xml:id="dbdoclet.50438216_10523">
713 <title><indexterm><primary>LNET</primary><secondary>testing</secondary></indexterm>Testing
714 the LNet Configuration</title>
715 <para>After configuring Lustre Networking, it is highly recommended
716 that you test your LNET configuration using the LNet Self-Test
717 provided with the Lustre software. For more information about using
718 LNET Self-Test, see <xref linkend="lnetselftest"/>.</para>
720 <section xml:id="dbdoclet.50438216_35668">
721 <title><indexterm><primary>LNET</primary><secondary>route
722 checker</secondary></indexterm>Configuring the Router Checker</title>
723 <para>In a Lustre configuration in which different types of networks,
724 such as a TCP/IP network and an Infiniband network, are connected by
725 routers, a router checker can be run on the clients and servers in the
726 routed configuration to monitor the status of the routers. In a
727 multi-hop routing configuration, router checkers can be configured on
728 routers to monitor the health of their next-hop routers.</para>
729 <para>A router checker is configured by setting lnet parameters in
730 <literal>lustre.conf</literal> by including an entry in this
733 <replaceable>router_checker_parameter</replaceable>=<replaceable>value</replaceable></screen>
734 <para>The router checker parameters are:</para>
737 <para><literal>live_router_check_interval</literal> - Specifies a
738 time interval in seconds after which the router checker will ping
739 the live routers. The default value is 0, meaning no checking is
740 done. To set the value to 60, enter:</para>
741 <screen>options lnet live_router_check_interval=60</screen>
744 <para><literal>dead_router_check_interval</literal> - Specifies a
745 time interval in seconds after which the router checker will check
746 for dead routers. The default value is 0, meaning no checking is
747 done. To set the value to 60, enter:</para>
748 <screen>options lnet dead_router_check_interval=60</screen>
751 <para>auto_down - Enables/disables (1/0) the automatic marking of
752 router state as up or down. The default value is 1. To disable
753 router marking, enter:</para>
754 <screen>options lnet auto_down=0</screen>
757 <para><literal>router_ping_timeout</literal> - Specifies a
758 timeout for the router checker when it checks live or dead
759 routers. The router checker sends a ping message to each dead or
760 live router once every dead_router_check_interval or
761 live_router_check_interval respectively. The default value is 50.
762 To set the value to 60, enter:</para>
763 <screen>options lnet router_ping_timeout=60</screen>
765 <para>The <literal>router_ping_timeout</literal> is consistent
766 with the default LND timeouts. You may have to increase it on very
767 large clusters if the LND timeout is also increased. For larger
768 clusters, we suggest increasing the check interval.</para>
772 <para><literal>check_routers_before_use</literal> - Specifies
773 that routers are to be checked before use. Set to off by
774 default. If this parameter is set to on, the
775 dead_router_check_interval parameter must be given a positive
776 integer value.</para>
777 <screen>options lnet check_routers_before_use=on</screen>
780 <para>The router checker obtains the following information from each router:</para>
783 <para> Time the router was disabled</para>
786 <para> Elapsed disable time</para>
789 <para>If the router checker does not get a reply message from the
790 router within router_ping_timeout seconds, it considers the router to
792 <para>If a router is marked 'up' and responds to a ping, the
793 timeout is reset.</para>
794 <para>If 100 packets have been sent successfully through a router, the
795 sent-packets counter for that router will have a value of 100.</para>
797 <section xml:id="dbdoclet.50438216_15200">
798 <title><indexterm><primary>LNET</primary><secondary>best
799 practice</secondary></indexterm>Best Practices for LNET
801 <para>For the <literal>networks</literal>, <literal>ip2nets</literal>,
802 and <literal>routes</literal> options, follow these best practices to
803 avoid configuration errors.</para>
805 <title><indexterm><primary>LNET</primary><secondary>escaping commas
806 with quotes</secondary></indexterm>Escaping commas with
808 <para>Depending on the Linux distribution, commas may need to be
809 escaped using single or double quotes. In the extreme case, the
810 <literal>options</literal> entry would look like this:</para>
811 <para><screen>options
812 lnet'networks="tcp0,elan0"'
813 'routes="tcp [2,10]@elan0"'</screen></para>
814 <para>Added quotes may confuse some distributions. Messages such as
815 the following may indicate an issue related to added quotes:</para>
816 <para><screen>lnet: Unknown parameter 'networks'</screen></para>
817 <para>A <literal>'Refusing connection - no matching
818 NID'</literal> message generally points to an error in the LNET
819 module configuration.</para>
822 <title><indexterm><primary>LNET</primary><secondary>comments</secondary></indexterm>Including
824 <para><emphasis>Place the semicolon terminating a comment
825 immediately after the comment.</emphasis> LNET silently ignores
826 everything between the <literal>#</literal> character at the
827 beginning of the comment and the next semicolon.</para>
828 <para>In this <emphasis>incorrect</emphasis> example, LNET silently
829 ignores <literal>pt11 192.168.0.[92,96]</literal>, resulting in
830 these nodes not being properly initialized. No error message is
832 <screen>options lnet ip2nets="pt10 192.168.0.[89,93]; # comment
833 with semicolon BEFORE comment \ pt11 192.168.0.[92,96];</screen>
834 <para>This <emphasis role="italic">correct</emphasis> example shows
835 the required syntax: </para>
836 <para><screen>options lnet ip2nets="pt10 192.168.0.[89,93] \
837 # comment with semicolon AFTER comment; \
838 pt11 192.168.0.[92,96] # comment</screen></para>
839 <para><emphasis role="italic">Do not add an excessive number of
840 comments.</emphasis> The Linux kernel limits the length of character
841 strings used in module options (usually to 1KB, but this may differ
842 between vendor kernels). If you exceed this limit, errors result and
843 the specified configuration may not be processed correctly.</para>