+ <section xml:id="dbdoclet.50438216_15201" condition='l27'>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>Configuring LNET</secondary>
+ </indexterm>Configuring LNET via <literal>lnetctl</literal></title>
+ <para>The <literal>lnetctl</literal> utility can be used to initialize
+ and configure the LNET kernel module after it has been loaded via
+ <literal>modprobe</literal>. In general the lnetctl format is as
+ follows: </para>
+ <screen>lnetctl cmd subcmd [options]</screen>
+ <para>The following configuration items are managed by the tool:</para>
+ <para>
+ <itemizedlist>
+ <listitem>
+ <para>Configuring/unconfiguring LNET</para>
+ </listitem>
+ <listitem>
+ <para>Adding/removing/showing Networks</para>
+ </listitem>
+ <listitem>
+ <para>Adding/removing/showing Routes</para>
+ </listitem>
+ <listitem>
+ <para>Enabling/Disabling routing</para>
+ </listitem>
+ <listitem>
+ <para>Configuring Router Buffer Pools</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Configuring LNET</title>
+ <para>After LNET has been loaded via <literal>modprobe</literal>,
+ <literal>lnetctl</literal> utility can be used to configure LNET
+ without bringing up networks which are specified in the module
+ parameters. It can also be used to configure network interfaces
+ specified in the module prameters by providing the
+ <literal>--all</literal> option.</para>
+ <screen>lnetctl lnet configure [--all]
+# --all: load NI configuration from module parameters</screen>
+ <para>The <literal>lnetctl</literal> utility can also be used to
+ unconfigure LNET.</para>
+ <screen>lnetctl lnet unconfigure</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Adding, Deleting and Showing networks</title>
+ <para>Networks can be added and deleted after the LNET kernel module
+ is loaded.</para>
+ <screen>lnetctl net add: add a network
+ --net: net name (ex tcp0)
+ --if: physical interface (ex eth0)
+ --peer_timeout: time to wait before declaring a peer dead
+ --peer_credits: defines the max number of inflight messages
+ --peer_buffer_credits: the number of buffer credits per peer
+ --credits: Network Interface credits
+ --cpts: CPU Partitions configured net uses
+ --help: display this help text
+
+Example:
+lnetctl net add --net tcp2 --if eth0
+ --peer_timeout 180 --peer_credits 8</screen>
+ <para>Networks can be deleted as shown below:</para>
+ <screen>net del: delete a network
+ --net: net name (ex tcp0)
+
+Example:
+lnetctl net del --net tcp2</screen>
+ <para>All or a subset of the configured networks can be shown. The
+ output can be non-verbose
+ or verbose.</para>
+ <screen>net show: show networks
+ --net: net name (ex tcp0) to filter on
+ --verbose: display detailed output per network
+
+Examples:
+lnetctl net show
+lnetctl net show --verbose
+lnetctl net show --net tcp2 --verbose</screen>
+ <para>Below are examples of non-detailed and detailed network
+ configuration show.</para>
+ <screen># non-detailed show
+> lnetctl net show --net tcp2
+net:
+ - nid: 192.168.205.130@tcp2
+ status: up
+ interfaces:
+ 0: eth3
+
+# detailed show
+> lnetctl net show --net tcp2 --verbose
+net:
+ - nid: 192.168.205.130@tcp2
+ status: up
+ interfaces:
+ 0: eth3
+ tunables:
+ peer_timeout: 180
+ peer_credits: 8
+ peer_buffer_credits: 0
+ credits: 256</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Adding, Deleting and Showing routes</title>
+ <para>A set of routes can be added to identify how LNET messages are
+ to be routed.</para>
+ <screen>lnetctl route add: add a route
+ --net: net name (ex tcp0) LNET message is destined to.
+ The can not be a local network.
+ --gateway: gateway node nid (ex 10.1.1.2@tcp) to route
+ all LNET messaged destined for the identified
+ network
+ --hop: number of hops to final destination
+ (1 < hops < 255)
+ --priority: priority of route (0 - highest prio)
+
+Example:
+lnetctl route add --net tcp2 --gateway 192.168.205.130@tcp1 --hop 2 --prio 1</screen>
+ <para>Routes can be deleted via the following <literal>lnetctl</literal> command.</para>
+ <screen>lnetctl route del: delete a route
+ --net: net name (ex tcp0)
+ --gateway: gateway nid (ex 10.1.1.2@tcp)
+
+Example:
+lnetctl route del --net tcp2 --gateway 192.168.205.130@tcp1</screen>
+ <para>Configured routes can be shown via the following
+ <literal>lnetctl</literal> command.</para>
+ <screen>lnetctl route show: show routes
+ --net: net name (ex tcp0) to filter on
+ --gateway: gateway nid (ex 10.1.1.2@tcp) to filter on
+ --hop: number of hops to final destination
+ (1 < hops < 255) to filter on
+ --priority: priority of route (0 - highest prio)
+ to filter on
+ --verbose: display detailed output per route
+
+Examples:
+# non-detailed show
+lnetctl route show
+
+# detailed show
+lnetctl route show --verbose</screen>
+ <para>When showing routes the <literal>--verbose</literal> option
+ outputs more detailed information. All show and error output are in
+ YAML format. Below are examples of both non-detailed and detailed
+ route show output.</para>
+ <screen>#Non-detailed output
+> lnetctl route show
+route:
+ - net: tcp2
+ gateway: 192.168.205.130@tcp1
+
+#detailed output
+> lnetctl route show --verbose
+route:
+ - net: tcp2
+ gateway: 192.168.205.130@tcp1
+ hop: 2
+ priority: 1
+ state: down</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Enabling and Disabling Routing</title>
+ <para>When an LNET node is configured as a router it will route LNet
+ messages not destined to itself. This feature can be enabled or
+ disabled as follows.</para>
+ <screen>lnetctl set routing [0 | 1]
+# 0 - disable routing feature
+# 1 - enable routing feature</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Showing routing information</title>
+ <para>When routing is enabled on a node, the tiny, small and large
+ routing buffers are allocated. See <xref
+ linkend="dbdoclet.50438272_73839"/> for more details on router
+ buffers. This information can be shown as follows:</para>
+ <screen>lnetctl routing show: show routing information
+
+Example:
+lnetctl routing show</screen>
+ <para>An example of the show output:</para>
+ <screen>> lnetctl routing show
+routing:
+ - cpt[0]:
+ tiny:
+ npages: 0
+ nbuffers: 2048
+ credits: 2048
+ mincredits: 2048
+ small:
+ npages: 1
+ nbuffers: 16384
+ credits: 16384
+ mincredits: 16384
+ large:
+ npages: 256
+ nbuffers: 1024
+ credits: 1024
+ mincredits: 1024
+ - enable: 1</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Configuring Routing Buffers</title>
+ <para> The routing buffers values configured specify the number of
+ buffers in each of the tiny, small and large groups.</para>
+ <para>It is often desirable to configure the tiny, small and large
+ routing buffers to some values other than the default. These values
+ are global values, when set they are used by all configured CPU
+ partitions. If routing is enabled then the values set take effect
+ immediately. If a larger number of buffers is specified, then
+ buffers are allocated to satisfy the configuration change. If fewer
+ buffers are configured then the excess buffers are freed as they
+ become unused. If routing is not set the values are not changed.
+ The buffer values are reset to default if routing is turned off and
+ on.</para>
+ <para>The <literal>lnetctl</literal> 'set' command can be
+ used to set these buffer values. A VALUE greater than 0
+ will set the number of buffers accordingly. A VALUE of 0
+ will reset the number of buffers to system defaults.</para>
+ <screen>set tiny_buffers:
+ set tiny routing buffers
+ VALUE must be greater than or equal to 0
+
+set small_buffers: set small routing buffers
+ VALUE must be greater than or equal to 0
+
+set large_buffers: set large routing buffers
+ VALUE must be greater than or equal to 0</screen>
+ <para>Usage examples:</para>
+ <screen>> lnetctl set tiny_buffers 4096
+> lnetctl set small_buffers 8192
+> lnetctl set large_buffers 2048</screen>
+ <para>The buffers can be set back to the default values as follows:</para>
+ <screen>> lnetctl set tiny_buffers 0
+> lnetctl set small_buffers 0
+> lnetctl set large_buffers 0</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Importing YAML Configuration File</title>
+ <para>Configuration can be described in YAML format and can be fed
+ into the <literal>lnetctl</literal> utility. The
+ <literal>lnetctl</literal> utility parses the YAML file and performs
+ the specified operation on all entities described there in. If no
+ operation is defined in the command as shown below, the default
+ operation is 'add'. The YAML syntax is described in a later
+ section.</para> <screen>lnetctl import FILE.yaml
+lnetctl import < FILE.yaml</screen>
+ <para>The '<literal>lnetctl</literal> import' command provides three
+ optional parameters to define the operation to be performed on the
+ configuration items described in the YAML file.</para>
+ <screen># if no options are given to the command the "add" command is assumed
+ # by default.
+lnetctl import --add FILE.yaml
+lnetctl import --add < FILE.yaml
+
+# to delete all items described in the YAML file
+lnetctl import --del FILE.yaml
+lnetctl import --del < FILE.yaml
+
+# to show all items described in the YAML file
+lnetctl import --show FILE.yaml
+lnetctl import --show < FILE.yaml</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Exporting Configuration in YAML format</title>
+ <para><literal>lnetctl</literal> utility provides the 'export'
+ command to dump current LNET configuration in YAML format </para>
+ <screen>lnetctl export FILE.yaml
+lnetctl export > FILE.yaml</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>cli</secondary>
+ </indexterm>Showing LNET Traffic Statistics</title>
+ <para><literal>lnetctl</literal> utility can dump the LNET traffic
+ statistiscs as follows</para>
+ <screen>lnetctl stats show</screen>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>yaml syntax</secondary>
+ </indexterm>YAML Syntax</title>
+ <para>The <literal>lnetctl</literal> utility can take in a YAML file
+ describing the configuration items that need to be operated on and
+ perform one of the following operations: add, delete or show on the
+ items described there in.</para>
+ <para>Net, routing and route YAML blocks are all defined as a YAML
+ sequence, as shown in the following sections. The stats YAML block
+ is a YAML object. Each sequence item can take a seq_no field. This
+ seq_no field is returned in the error block. This allows the caller
+ to associate the error with the item that caused the error. The
+ <literal>lnetctl</literal> utilty does a best effort at configuring
+ items defined in the YAML file. It does not stop processing the file
+ at the first error.</para>
+ <para>Below is the YAML syntax describing the various
+ configuration elements which can be operated on via DLC. Not all
+ YAML elements are requied for all operations (add/delete/show).
+ The system ignores elements which are not pertinent to the requested
+ operation.</para>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>network yaml syntax</secondary>
+ </indexterm>Network Configuration</title>
+ <para/>
+ <screen>net:
+ - net: <network. Ex: tcp or o2ib>
+ interfaces:
+ 0: <physical interface>
+ detail: <This is only applicable for show command. 1 - output detailed info. 0 - basic output>
+ tunables:
+ peer_timeout: <Integer. Timeout before consider a peer dead>
+ peer_credits: <Integer. Transmit credits for a peer>
+ peer_buffer_credits: <Integer. Credits available for receiving messages>
+ credits: <Integer. Network Interface credits>
+ SMP: <An array of integers of the form: "[x,y,...]", where each
+ integer represents the CPT to associate the network interface
+ with> seq_no: <integer. Optional. User generated, and is
+ passed back in the YAML error block></screen>
+ <para>Both seq_no and detail fields do not appear in the show output.</para>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>buffer yaml syntax</secondary>
+ </indexterm>Enable Routing and Adjust Router Buffer Configuration</title>
+ <para/>
+ <screen>routing:
+ - tiny: <Integer. Tiny buffers>
+ small: <Integer. Small buffers>
+ large: <Integer. Large buffers>
+ enable: <0 - disable routing. 1 - enable routing>
+ seq_no: <Integer. Optional. User generated, and is passed back in the YAML error block></screen>
+ <para>The seq_no field does not appear in the show output</para>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>statistics yaml syntax</secondary>
+ </indexterm>Show Statistics</title>
+ <para/>
+ <screen>statistics:
+ seq_no: <Integer. Optional. User generated, and is passed back in the YAML error block></screen>
+ <para>The seq_no field does not appear in the show output</para>
+ </section>
+ <section>
+ <title><indexterm>
+ <primary>LNET</primary>
+ <secondary>router yaml syntax</secondary>
+ </indexterm>Route Configuration</title>
+ <para/>
+ <screen>route:
+ - net: <network. Ex: tcp or o2ib>
+ gateway: <nid of the gateway in the form <ip>@<net>: Ex: 192.168.29.1@tcp>
+ hop: <an integer between 1 and 255. Optional>
+ detail: <This is only applicable for show commands. 1 - output detailed info. 0. basic output>
+ seq_no: <integer. Optional. User generated, and is passed back in the YAML error block></screen>
+ <para>Both seq_no and detail fields do not appear in the show output.</para>
+ </section>
+ </section>
+ </section>