<para> OFED Version 2</para>
</entry>
</row>
- <row>
- <entry>
- <para> mx</para>
- </entry>
- <entry>
- <para> Myrinet MX</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> gm</para>
- </entry>
- <entry>
- <para> Myrinet GM-2</para>
- </entry>
- </row>
</tbody>
</tgroup>
</informaltable>
[ <w> ]
<net-spec> :== <network> [ "(" <interface-list> ")" ]
<network> :== <nettype> [ <number> ]
-<nettype> :== "tcp" | "elan" | "openib" | ...
+<nettype> :== "tcp" | "elan" | "o2ib" | ...
<iface-list> :== <interface> [ "," <iface-list> ]
<ip-range> :== <r-expr> "." <r-expr> "." <r-expr> "." <r-expr>
<r-expr> :== <number> | "*" | "[" <r-list> "]"
<net-sep> :== ";" | "\n"
<w> :== <whitespace-chars> { <whitespace-chars> }
</screen>
- <para><literal><net-spec></literal> contains enough information to uniquely identify the network and load an appropriate LND. The LND determines the missing "address-within-network" part of the NID based on the interfaces it can use.</para>
- <para><literal><iface-list></literal> specifies which hardware interface the network can use. If omitted, all interfaces are used. LNDs that do not support the <literal><iface-list></literal> syntax cannot be configured to use particular interfaces and just use what is there. Only a single instance of these LNDs can exist on a node at any time, and <literal><iface-list></literal> must be omitted.</para>
- <para><literal><net-match></literal> entries are scanned in the order declared to see if one of the node's IP addresses matches one of the <literal><ip-range></literal> expressions. If there is a match, <literal><net-spec></literal> specifies the network to instantiate. Note that it is the first match for a particular network that counts. This can be used to simplify the match expression for the general case by placing it after the special cases. For example:</para>
+ <para><literal><net-spec></literal> contains enough information to
+ uniquely identify the network and load an appropriate LND. The LND
+ determines the missing "address-within-network" part of the
+ NID based on the interfaces it can use.</para>
+ <para><literal><iface-list></literal> specifies which hardware
+ interface the network can use. If omitted, all interfaces are used. LNDs
+ that do not support the <literal><iface-list></literal> syntax
+ cannot be configured to use particular interfaces and just use what is
+ there. Only a single instance of these LNDs can exist on a node at any
+ time, and <literal><iface-list></literal> must be omitted.</para>
+ <para><literal><net-match></literal> entries are scanned in the
+ order declared to see if one of the node's IP addresses matches one
+ of the <literal><ip-range></literal> expressions. If there is a
+ match, <literal><net-spec></literal> specifies the network to
+ instantiate. Note that it is the first match for a particular network
+ that counts. This can be used to simplify the match expression for the
+ general case by placing it after the special cases. For example:</para>
<screen>ip2nets="tcp(eth1,eth2) 134.32.1.[4-10/2]; tcp(eth1) *.*.*.*"</screen>
- <para>4 nodes on the 134.32.1.* network have 2 interfaces (134.32.1.{4,6,8,10}) but all the rest have 1.</para>
- <screen>ip2nets="<emphasis role="bold">vib</emphasis> 192.168.0.*; tcp(eth2) 192.168.0.[1,7,4,12]" </screen>
- <para>This describes an IB cluster on 192.168.0.*. Four of these nodes also have IP interfaces; these four could be used as routers.</para>
- <para>Note that match-all expressions (For instance, <literal>*.*.*.*</literal>) effectively mask all other</para>
- <para> <literal><net-match></literal> entries specified after them. They should be used with caution.</para>
- <para>Here is a more complicated situation, the route parameter is explained below. We have:</para>
+ <para>4 nodes on the 134.32.1.* network have 2 interfaces
+ (134.32.1.{4,6,8,10}) but all the rest have 1.</para>
+ <screen>ip2nets="<emphasis role="bold">o2ib</emphasis> 192.168.0.*; tcp(eth2) 192.168.0.[1,7,4,12]" </screen>
+ <para>This describes an IB cluster on 192.168.0.*. Four of these nodes
+ also have IP interfaces; these four could be used as routers.</para>
+ <para>Note that match-all expressions (For instance,
+ <literal>*.*.*.*</literal>) effectively mask all other</para>
+ <para> <literal><net-match></literal> entries specified after
+ them. They should be used with caution.</para>
+ <para>Here is a more complicated situation, the route parameter is
+ explained below. We have:</para>
<itemizedlist>
<listitem>
<para>Two TCP subnets</para>
<para>One Elan subnet</para>
</listitem>
<listitem>
- <para>One machine set up as a router, with both TCP and Elan interfaces</para>
+ <para>One machine set up as a router, with both TCP and Elan
+ interfaces</para>
</listitem>
<listitem>
- <para>IP over Elan configured, but only IP will be used to label the nodes.</para>
+ <para>IP over Elan configured, but only IP will be used to label the
+ nodes.</para>
</listitem>
</itemizedlist>
<screen>options lnet ip2nets=â€tcp 198.129.135.* 192.128.88.98; \
elan 198.128.88.98@tcp # TCP NID of router '</screen>
</section>
<section remap="h4">
- <title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>tcp</tertiary></indexterm>
-networks ("tcp")</title>
- <para>This is an alternative to "<literal>ip2nets</literal>" which can be used to specify the networks to be instantiated explicitly. The syntax is a simple comma separated list of <literal><net-spec></literal>s (see above). The default is only used if neither 'ip2nets' nor 'networks' is specified.</para>
+ <title><indexterm><primary>configuring</primary>
+ <secondary>network</secondary><tertiary>tcp</tertiary></indexterm>
+ networks ("tcp")</title>
+ <para>This is an alternative to "<literal>ip2nets</literal>"
+ which can be used to specify the networks to be instantiated explicitly.
+ The syntax is a simple comma separated list of
+ <literal><net-spec></literal>s (see above). The default is only
+ used if neither 'ip2nets' nor 'networks' is
+ specified.</para>
</section>
<section remap="h4">
<title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>routes</tertiary></indexterm>
<entry> :== <numeric range> | <non-numeric item>
<numeric range> :== <number> [ "-" <number> [ "/" <number> ] ]</screen>
<para>The expansion is a list enclosed in square brackets. Numeric items in the list may be a single number, a contiguous range of numbers, or a strided range of numbers. For example, <literal>routes="elan 192.168.1.[22-24]@tcp"</literal> says that network <literal>elan0</literal> is adjacent (hopcount defaults to 1); and is accessible via 3 routers on the <literal>tcp0</literal> network (<literal>192.168.1.22@tcp</literal>, <literal>192.168.1.23@tcp</literal> and <literal>192.168.1.24@tcp</literal>).</para>
- <para><literal>routes="[tcp,vib] 2 [8-14/2]@elan"</literal> says that 2 networks (<literal>tcp0</literal> and <literal>vib0</literal>) are accessible through 4 routers (<literal>8@elan</literal>, <literal>10@elan</literal>, <literal>12@elan</literal> and <literal>14@elan</literal>). The hopcount of 2 means that traffic to both these networks will be traversed 2 routers - first one of the routers specified in this entry, then one more.</para>
+ <para><literal>routes="[tcp,o2ib] 2 [8-14/2]@elan"</literal>
+says that 2 networks (<literal>tcp0</literal> and <literal>o2ib0</literal>) are accessible through 4 routers (<literal>8@elan</literal>, <literal>10@elan</literal>, <literal>12@elan</literal> and <literal>14@elan</literal>). The hopcount of 2 means that traffic to both these networks will be traversed 2 routers - first one of the routers specified in this entry, then one more.</para>
<para>Duplicate entries, entries that route to a local network, and entries that specify routers on a non-local network are ignored.</para>
<para>Prior to release 2.5, a conflict between equivalent entries was resolved in favor of the route with the shorter hopcount. The hopcount, if omitted, defaults to 1 (the remote network is adjacent)..</para>
<para condition='l25'>Since 2.5, equivalent entries are resolved in favor of the route with the lowest priority number or shorter hopcount if the priorities are equal. The priority, if omitted, defaults to 0. The hopcount, if omitted, defaults to 1 (the remote network is adjacent).</para>
<section>
<title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>rnet_htable_size</tertiary></indexterm>
<literal>rnet_htable_size</literal></title>
- <para condition='l23'><literal>rnet_htable_size</literal> is an integer that indicates how many remote networks the internal LNet hash table is configured to handle. <literal>rnet_htable_size</literal> is used for optimizing the hash table size and does not put a limit on how many remote networks you can have. The default hash table size when this parameter is not specified is: 128.</para>
+ <para><literal>rnet_htable_size</literal> is an integer that indicates how many remote networks the internal LNet hash table is configured to handle. <literal>rnet_htable_size</literal> is used for optimizing the hash table size and does not put a limit on how many remote networks you can have. The default hash table size when this parameter is not specified is: 128.</para>
</section>
</section>
<section remap="h3" xml:id="section_ngq_qhy_zl">
parameter, all non-loopback IP interfaces are used. The address-within-network is determined
by the address of the first IP interface an instance of the <literal>socklnd</literal>
encounters.</para>
- <para>Consider a node on the 'edge' of an InfiniBand network, with a low-bandwidth
- management Ethernet (<literal>eth0</literal>), IP over IB configured
- (<literal>ipoib0</literal>), and a pair of GigE NICs
- (<literal>eth1</literal>,<literal>eth2</literal>) providing off-cluster connectivity. This
- node should be configured with '<literal>networks=vib,tcp(eth1,eth2)</literal>' to
- ensure that the <literal>socklnd</literal> ignores the management Ethernet and IPoIB.</para>
+ <para>Consider a node on the 'edge' of an InfiniBand network,
+ with a low-bandwidth management Ethernet (<literal>eth0</literal>), IP
+ over IB configured (<literal>ipoib0</literal>), and a pair of GigE NICs
+ (<literal>eth1</literal>,<literal>eth2</literal>) providing off-cluster
+ connectivity. This node should be configured with '
+ <literal>networks=o2ib,tcp(eth1,eth2)</literal>' to ensure that the
+ <literal>socklnd</literal> ignores the management Ethernet and IPoIB.
+ </para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
</tgroup>
</informaltable>
</section>
- <section remap="h3">
- <title><indexterm><primary>configuring</primary><secondary>MX LND</secondary></indexterm>MX LND</title>
- <para><literal>MXLND</literal> supports a number of load-time parameters using Linux's module parameter system. The following variables are available:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry>
- <para><emphasis role="bold">Variable</emphasis></para>
- </entry>
- <entry>
- <para><emphasis role="bold">Description</emphasis></para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry>
- <para> <literal>n_waitd</literal></para>
- </entry>
- <entry>
- <para>Number of completion daemons.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>max_peers</literal></para>
- </entry>
- <entry>
- <para>Maximum number of peers that may connect.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>cksum</literal></para>
- </entry>
- <entry>
- <para>Enables small message (below 4 KB) checksums if set to a non-zero value.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>ntx</literal></para>
- </entry>
- <entry>
- <para>Number of total tx message descriptors.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>credits</literal></para>
- </entry>
- <entry>
- <para>Number of concurrent sends to a single peer.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>board</literal></para>
- </entry>
- <entry>
- <para>Index value of the Myrinet board (NIC).</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>ep_id</literal></para>
- </entry>
- <entry>
- <para>MX endpoint ID.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>polling</literal></para>
- </entry>
- <entry>
- <para>Use zero (0) to block (wait). A value greater than 0 will poll that many times before blocking.</para>
- </entry>
- </row>
- <row>
- <entry>
- <para> <literal>hosts</literal></para>
- </entry>
- <entry>
- <para>IP-to-hostname resolution file.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para>Of the described variables, only hosts is required. It must be the absolute path to the MXLND hosts file.</para>
- <para>For example:</para>
- <screen>options kmxlnd hosts=/etc/hosts.mxlnd</screen>
- <para>The file format for the hosts file is:</para>
- <screen>IP HOST BOARD EP_ID</screen>
- <para>The values must be space and/or tab separated where:</para>
- <para><literal>IP</literal> is a valid IPv4 address</para>
- <para><literal>HOST</literal> is the name returned by <literal>`hostname`</literal> on that machine</para>
- <para><literal>BOARD</literal> is the index of the Myricom NIC (0 for the first card, etc.)</para>
- <para><literal>EP_ID</literal> is the MX endpoint ID</para>
- <para>To obtain the optimal performance for your platform, you may want to vary the remaining options.</para>
- <para><literal>n_waitd(1)</literal> sets the number of threads that process completed MX requests (sends and receives).</para>
- <para><literal>max_peers(1024)</literal> tells MXLND the upper limit of machines that it will need to communicate with. This affects how many receives it will pre-post and each receive will use one page of memory. Ideally, on clients, this value will be equal to the total number of Lustre servers (MDS and OSS). On servers, it needs to equal the total number of machines in the storage system. cksum (0) turns on small message checksums. It can be used to aid in troubleshooting. MX also provides an optional checksumming feature which can check all messages (large and small). For details, see the MX README.</para>
- <para><literal>ntx(256)</literal> is the number of total sends in flight from this machine. In actuality, MXLND reserves half of them for connect messages so make this value twice as large as you want for the total number of sends in flight.</para>
- <para><literal>credits(8)</literal> is the number of in-flight messages for a specific peer.
- This is part of the flow-control system in provided by the Lustre software. Increasing this
- value may improve performance but it requires more memory because each message requires at
- least one page.</para>
- <para><literal>board(0)</literal> is the index of the Myricom NIC. Hosts can have multiple Myricom NICs and this identifies which one MXLND should use. This value must match the board value in your MXLND hosts file for this host.</para>
- <para><literal>ep_id(3)</literal> is the MX endpoint ID. Each process that uses MX is required to have at least one MX endpoint to access the MX library and NIC. The ID is a simple index starting at zero (0). This value must match the endpoint ID value in your MXLND hosts file for this host.</para>
- <para><literal>polling(0)</literal> determines whether this host will poll or block for MX request completions. A value of 0 blocks and any positive value will poll that many times before blocking. Since polling increases CPU usage, we suggest that you set this to zero (0) on the client and experiment with different values for servers.</para>
- </section>
</section>
</chapter>
git://git.whamcloud.com / doc / manual.git / blobdiff