+ <section condition='l2A' xml:id="lnet_config.manual_addshowdelete">
+ <title><indexterm>
+ <primary>LNet</primary>
+ <secondary>cli</secondary>
+ </indexterm>Manual Adding, Deleting and Showing Peers</title>
+ <para>The <emphasis role="bold"><literal>lnetctl peer add</literal>
+ </emphasis> command is used to manually add a remote peer to a software
+ multi-rail configuration. For the dynamic peer discovery capability
+ introduced in Lustre Release 2.11.0, please see
+ <xref linkend="lnet_config.dynamic_discovery" />.</para>
+ <para>When configuring peers, use the <literal>–-prim_nid</literal>
+ option to specify the key or primary nid of the peer node. Then
+ follow that with the <literal>--nid</literal> option to specify a
+ set of comma separated NIDs.</para>
+ <screen>peer add: add a peer
+ --prim_nid: primary NID of the peer
+ --nid: comma separated list of peer nids (e.g. 10.1.1.2@tcp0)
+ --non_mr: if specified this interface is created as a non mulit-rail
+ capable peer. Only one NID can be specified in this case.</screen>
+ <para>For example:</para>
+ <screen>
+ lnetctl peer add --prim_nid 10.10.10.2@tcp --nid 10.10.3.3@tcp1,10.4.4.5@tcp2
+ </screen>
+ <para>The <literal>--prim-nid</literal> (primary nid for the peer
+ node) can go unspecified. In this case, the first listed NID in the
+ <literal>--nid</literal> option becomes the primary nid of the peer.
+ For example:</para>
+ <screen>
+ lnetctl peer_add --nid 10.10.10.2@tcp,10.10.3.3@tcp1,10.4.4.5@tcp2</screen>
+ <para>YAML can also be used to configure peers:</para>
+ <screen>peer:
+ - primary nid: <key or primary nid>
+ Multi-Rail: True
+ peer ni:
+ - nid: <nid 1>
+ - nid: <nid 2>
+ - nid: <nid n></screen>
+ <para>As with all other commands, the result of the
+ <literal>lnetctl peer show</literal> command can be used to gather
+ information to aid in configuring or deleting a peer:</para>
+ <screen>lnetctl peer show -v</screen>
+ <para>Example output from the <literal>lnetctl peer show</literal>
+ command:</para>
+ <screen>peer:
+ - primary nid: 192.168.122.218@tcp
+ Multi-Rail: True
+ peer ni:
+ - nid: 192.168.122.218@tcp
+ state: NA
+ max_ni_tx_credits: 8
+ available_tx_credits: 8
+ available_rtr_credits: 8
+ min_rtr_credits: -1
+ tx_q_num_of_buf: 0
+ send_count: 6819
+ recv_count: 6264
+ drop_count: 0
+ refcount: 1
+ - nid: 192.168.122.78@tcp
+ state: NA
+ max_ni_tx_credits: 8
+ available_tx_credits: 8
+ available_rtr_credits: 8
+ min_rtr_credits: -1
+ tx_q_num_of_buf: 0
+ send_count: 7061
+ recv_count: 6273
+ drop_count: 0
+ refcount: 1
+ - nid: 192.168.122.96@tcp
+ state: NA
+ max_ni_tx_credits: 8
+ available_tx_credits: 8
+ available_rtr_credits: 8
+ min_rtr_credits: -1
+ tx_q_num_of_buf: 0
+ send_count: 6939
+ recv_count: 6286
+ drop_count: 0
+ refcount: 1</screen>
+ <para>Use the following <literal>lnetctl</literal> command to delete a
+ peer:</para>
+ <screen>peer del: delete a peer
+ --prim_nid: Primary NID of the peer
+ --nid: comma separated list of peer nids (e.g. 10.1.1.2@tcp0)</screen>
+ <para><literal>prim_nid</literal> should always be specified. The
+ <literal>prim_nid</literal> identifies the peer. If the
+ <literal>prim_nid</literal> is the only one specified, then the
+ entire peer is deleted.</para>
+ <para>Example of deleting a single nid of a peer (10.10.10.3@tcp):
+ </para>
+ <screen>lnetctl peer del --prim_nid 10.10.10.2@tcp --nid 10.10.10.3@tcp</screen>
+ <para>Example of deleting the entire peer:</para>
+ <screen>lnetctl peer del --prim_nid 10.10.10.2@tcp</screen>
+ </section>
+ <section condition='l2B' xml:id="lnet_config.dynamic_discovery">
+ <title><indexterm>
+ <primary>LNet</primary>
+ <secondary>cli</secondary>
+ <tertiary>dynamic discovery</tertiary>
+ </indexterm>Dynamic Peer Discovery</title>
+ <section xml:id="lnet_config.dynamic_discovery.overview">
+ <title>Overview</title>
+ <para>Dynamic Discovery (DD) is a feature that allows nodes to
+ dynamically discover a peer's interfaces without having to explicitly
+ configure them. This is very useful for Multi-Rail (MR)
+ configurations. In large clusters, there could be hundreds of nodes
+ and having to configure MR peers on each node becomes error prone.
+ Dynamic Discovery is enabled by default and uses a new protocol based
+ on LNet pings to discover the interfaces of the remote peers on first
+ message.</para>
+ </section>
+ <section xml:id="lnet_config.dynamic_discovery.protocol">
+ <title>Protocol</title>
+ <para>When LNet on a node is requested to send a message to a peer it
+ first attempts to ping the peer. The reply to the ping contains the
+ peer's NIDs as well as a feature bit outlining what the peer supports.
+ Dynamic Discovery adds a Multi-Rail feature bit. If the peer is
+ Multi-Rail capable, it sets the MR bit in the ping reply. When the
+ node receives the reply it checks the MR bit, and if it is set it then
+ pushes its own list of NIDs to the peer using a new PUT message,
+ referred to as a "push ping". After this brief protocol, both the peer
+ and the node will have each other's list of interfaces. The MR
+ algorithm can then proceed to use the list of interfaces of the
+ corresponding peer.</para>
+ <para>If the peer is not MR capable, it will not set the MR feature
+ bit in the ping reply. The node will understand that the peer is
+ not MR capable and will only use the interface provided by upper
+ layers for sending messages.</para>
+ </section>
+ <section xml:id="lnet_config.dynamic_discovery.userspace_config">
+ <title>Dynamic Discovery and User-space Configuration</title>
+ <para>It is possible to configure the peer manually while Dynamic
+ Discovery is running. Manual peer configuration always takes precedence
+ over Dynamic Discovery. If there is a discrepancy between the manual
+ configuration and the dynamically discovered information, a warning is
+ printed.</para>
+ </section>
+ <section xml:id="lnet_config.dynamic_discovery.config">
+ <title>Configuration</title>
+ <para>Dynamic Discovery is very light on the configuration side. It can
+ only be turned on or turned off. To turn the feature on or off, the
+ following command is used:</para>
+ <screen>lnetctl set discovery [0 | 1]</screen>
+ <para>To check the current <literal>discovery</literal> setting, the
+ <literal>lnetctl global show</literal> command can be used as shown in
+ <xref linkend="lnet_config.show_global_settings"/>.</para>
+ </section>
+ <section xml:id="lnet_config.dynamic_discovery.ondemand">
+ <title>Initiating Dynamic Discovery on Demand</title>
+ <para>It is possible to initiate the Dynamic Discovery protocol on demand
+ without having to wait for a message to be sent to the peer. This can
+ be done with the following command:</para>
+ <screen>lnetctl discover <peer_nid> [<peer_nid> ...]</screen>
+ </section>
+ </section>