<para>A separate <literal>lustre.conf</literal> file makes distributing the configuration much easier.</para>
</listitem>
<listitem>
- <para>If you set <literal>config_on_load=1</literal>, LNET starts at <literal>modprobe</literal> time rather than waiting for Lustre to start. This ensures routers start working at module load time.</para>
+ <para>If you set <literal>config_on_load=1</literal>, LNET starts at
+ <literal>modprobe</literal> time rather than waiting for the Lustre file system to
+ start. This ensures routers start working at module load time.</para>
</listitem>
</itemizedlist>
<screen># lctl
</tgroup>
</informaltable>
<note>
- <para>Lustre ignores the loopback interface (<literal>lo0</literal>), but Lustre use any IP addresses aliased to the loopback (by default). When in doubt, explicitly specify networks.</para>
+ <para>The Lustre software ignores the loopback interface (<literal>lo0</literal>), but the
+ Lustre file system uses any IP addresses aliased to the loopback (by default). When in
+ doubt, explicitly specify networks.</para>
</note>
<para><literal>ip2nets</literal> ("") is a string that lists globally-available networks, each with a set of IP address ranges. LNET determines the locally-available networks from this list by matching the IP address ranges with the local IPs of a node. The purpose of this option is to be able to use the same <literal>modules.conf</literal> file across a variety of nodes on different networks. The string has the following syntax.</para>
<screen><ip2nets> :== <net-match> [ <comment> ] { <net-sep> <net-match> }
<para><literal>routes="[tcp,vib] 2 [8-14/2]@elan"</literal> says that 2 networks (<literal>tcp0</literal> and <literal>vib0</literal>) are accessible through 4 routers (<literal>8@elan</literal>, <literal>10@elan</literal>, <literal>12@elan</literal> and <literal>14@elan</literal>). The hopcount of 2 means that traffic to both these networks will be traversed 2 routers - first one of the routers specified in this entry, then one more.</para>
<para>Duplicate entries, entries that route to a local network, and entries that specify routers on a non-local network are ignored.</para>
<para>Prior to release 2.5, a conflict between equivalent entries was resolved in favor of the route with the shorter hopcount. The hopcount, if omitted, defaults to 1 (the remote network is adjacent)..</para>
- <para condition='125'>Since 2.5, equivalent entries are resolved in favor of the route with the lowest priority number or shorter hopcount if the priorities are equal. The priority, if omitted, defaults to 0. The hopcount, if omitted, defaults to 1 (the remote network is adjacent).</para>
+ <para condition='l25'>Since 2.5, equivalent entries are resolved in favor of the route with the lowest priority number or shorter hopcount if the priorities are equal. The priority, if omitted, defaults to 0. The hopcount, if omitted, defaults to 1 (the remote network is adjacent).</para>
<para>It is an error to specify routes to the same destination with routers on different local networks.</para>
<para>If the target network string contains no expansions, then the hopcount defaults to 1 and may be omitted (that is, the remote network is adjacent). In practice, this is true for most multi-network configurations. It is an error to specify an inconsistent hop count for a given target network. This is why an explicit hopcount is required if the target network string specifies more than one network.</para>
</section>
<section>
<title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>rnet_htable_size</tertiary></indexterm>
<literal>rnet_htable_size</literal></title>
- <para condition='123'><literal>rnet_htable_size</literal> is an integer that indicates how many remote networks the internal LNet hash table is configured to handle. <literal>rnet_htable_size</literal> is used for optimizing the hash table size and does not put a limit on how many remote networks you can have. The default hash table size when this parameter is not specified is: 128.</para>
+ <para condition='l23'><literal>rnet_htable_size</literal> is an integer that indicates how many remote networks the internal LNet hash table is configured to handle. <literal>rnet_htable_size</literal> is used for optimizing the hash table size and does not put a limit on how many remote networks you can have. The default hash table size when this parameter is not specified is: 128.</para>
</section>
</section>
- <section remap="h3">
- <title><indexterm><primary>configuring</primary><secondary>network</secondary><tertiary>SOCKLND</tertiary></indexterm>
-<literal>SOCKLND</literal> Kernel TCP/IP LND</title>
- <para>The <literal>SOCKLND</literal> kernel TCP/IP LND (<literal>socklnd</literal>) is connection-based and uses the acceptor to establish communications via sockets with its peers.</para>
- <para>It supports multiple instances and load balances dynamically over multiple interfaces. If no interfaces are specified by the <literal>ip2nets</literal> or networks module parameter, all non-loopback IP interfaces are used. The address-within-network is determined by the address of the first IP interface an instance of the <literal>socklnd</literal> encounters.</para>
- <para>Consider a node on the 'edge' of an InfiniBand network, with a low-bandwidth management Ethernet (<literal>eth0</literal>), IP over IB configured (<literal>ipoib0</literal>), and a pair of GigE NICs (<literal>eth1</literal>,<literal>eth2</literal>) providing off-cluster connectivity. This node should be configured with '<literal>networks=vib,tcp(eth1,eth2)</literal>' to ensure that the <literal>socklnd</literal> ignores the management Ethernet and IPoIB.</para>
+ <section remap="h3" xml:id="section_ngq_qhy_zl">
+ <title><indexterm>
+ <primary>configuring</primary>
+ <secondary>network</secondary>
+ <tertiary>SOCKLND</tertiary>
+ </indexterm>
+ <literal>SOCKLND</literal> Kernel TCP/IP LND</title>
+ <para>The <literal>SOCKLND</literal> kernel TCP/IP LND (<literal>socklnd</literal>) is
+ connection-based and uses the acceptor to establish communications via sockets with its
+ peers.</para>
+ <para>It supports multiple instances and load balances dynamically over multiple interfaces.
+ If no interfaces are specified by the <literal>ip2nets</literal> or networks module
+ parameter, all non-loopback IP interfaces are used. The address-within-network is determined
+ by the address of the first IP interface an instance of the <literal>socklnd</literal>
+ encounters.</para>
+ <para>Consider a node on the 'edge' of an InfiniBand network, with a low-bandwidth
+ management Ethernet (<literal>eth0</literal>), IP over IB configured
+ (<literal>ipoib0</literal>), and a pair of GigE NICs
+ (<literal>eth1</literal>,<literal>eth2</literal>) providing off-cluster connectivity. This
+ node should be configured with '<literal>networks=vib,tcp(eth1,eth2)</literal>' to
+ ensure that the <literal>socklnd</literal> ignores the management Ethernet and IPoIB.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<tbody>
<row>
<entry>
- <para> <literal>timeout</literal></para>
- <para> <literal>(50,W)</literal></para>
+ <para>
+ <literal>timeout</literal></para>
+ <para>
+ <literal>(50,W)</literal></para>
</entry>
<entry>
- <para>Time (in seconds) that communications may be stalled before the LND completes them with failure.</para>
+ <para>Time (in seconds) that communications may be stalled before the LND completes
+ them with failure.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>nconnds</literal></para>
- <para> <literal>(4)</literal></para>
+ <para>
+ <literal>nconnds</literal></para>
+ <para>
+ <literal>(4)</literal></para>
</entry>
<entry>
<para>Sets the number of connection daemons.</para>
</row>
<row>
<entry>
- <para> <literal>min_reconnectms</literal></para>
- <para> <literal>(1000,W)</literal></para>
+ <para>
+ <literal>min_reconnectms</literal></para>
+ <para>
+ <literal>(1000,W)</literal></para>
</entry>
<entry>
- <para>Minimum connection retry interval (in milliseconds). After a failed connection attempt, this is the time that must elapse before the first retry. As connections attempts fail, this time is doubled on each successive retry up to a maximum of '<literal>max_reconnectms</literal>'.</para>
+ <para>Minimum connection retry interval (in milliseconds). After a failed connection
+ attempt, this is the time that must elapse before the first retry. As connections
+ attempts fail, this time is doubled on each successive retry up to a maximum of
+ '<literal>max_reconnectms</literal>'.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>max_reconnectms</literal></para>
- <para> <literal>(6000,W)</literal></para>
+ <para>
+ <literal>max_reconnectms</literal></para>
+ <para>
+ <literal>(6000,W)</literal></para>
</entry>
<entry>
<para>Maximum connection retry interval (in milliseconds).</para>
</row>
<row>
<entry>
- <para> <literal>eager_ack</literal></para>
- <para> <literal>(0 on linux,</literal></para>
- <para> <literal>1 on darwin,W)</literal></para>
+ <para>
+ <literal>eager_ack</literal></para>
+ <para>
+ <literal>(0 on linux,</literal></para>
+ <para>
+ <literal>1 on darwin,W)</literal></para>
</entry>
<entry>
- <para>Boolean that determines whether the <literal>socklnd</literal> should attempt to flush sends on message boundaries.</para>
+ <para>Boolean that determines whether the <literal>socklnd</literal> should attempt
+ to flush sends on message boundaries.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>typed_conns</literal></para>
- <para> <literal>(1,Wc)</literal></para>
+ <para>
+ <literal>typed_conns</literal></para>
+ <para>
+ <literal>(1,Wc)</literal></para>
</entry>
<entry>
- <para>Boolean that determines whether the <literal>socklnd</literal> should use different sockets for different types of messages. When clear, all communication with a particular peer takes place on the same socket. Otherwise, separate sockets are used for bulk sends, bulk receives and everything else.</para>
+ <para>Boolean that determines whether the <literal>socklnd</literal> should use
+ different sockets for different types of messages. When clear, all communication
+ with a particular peer takes place on the same socket. Otherwise, separate sockets
+ are used for bulk sends, bulk receives and everything else.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>min_bulk</literal></para>
- <para> <literal>(1024,W)</literal></para>
+ <para>
+ <literal>min_bulk</literal></para>
+ <para>
+ <literal>(1024,W)</literal></para>
</entry>
<entry>
<para>Determines when a message is considered "bulk".</para>
</row>
<row>
<entry>
- <para> <literal>tx_buffer_size, rx_buffer_size</literal></para>
- <para> <literal>(8388608,Wc)</literal></para>
+ <para>
+ <literal>tx_buffer_size, rx_buffer_size</literal></para>
+ <para>
+ <literal>(8388608,Wc)</literal></para>
</entry>
<entry>
- <para>Socket buffer sizes. Setting this option to zero (0), allows the system to auto-tune buffer sizes. </para>
+ <para>Socket buffer sizes. Setting this option to zero (0), allows the system to
+ auto-tune buffer sizes. </para>
<warning>
- <para>Be very careful changing this value as improper sizing can harm performance.</para>
+ <para>Be very careful changing this value as improper sizing can harm
+ performance.</para>
</warning>
</entry>
</row>
<row>
<entry>
- <para> <literal>nagle</literal></para>
- <para> <literal>(0,Wc)</literal></para>
+ <para>
+ <literal>nagle</literal></para>
+ <para>
+ <literal>(0,Wc)</literal></para>
</entry>
<entry>
- <para>Boolean that determines if <literal>nagle</literal> should be enabled. It should never be set in production systems.</para>
+ <para>Boolean that determines if <literal>nagle</literal> should be enabled. It
+ should never be set in production systems.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>keepalive_idle</literal></para>
- <para> <literal>(30,Wc)</literal></para>
+ <para>
+ <literal>keepalive_idle</literal></para>
+ <para>
+ <literal>(30,Wc)</literal></para>
</entry>
<entry>
- <para>Time (in seconds) that a socket can remain idle before a keepalive probe is sent. Setting this value to zero (0) disables keepalives.</para>
+ <para>Time (in seconds) that a socket can remain idle before a keepalive probe is
+ sent. Setting this value to zero (0) disables keepalives.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>keepalive_intvl</literal></para>
- <para> <literal>(2,Wc)</literal></para>
+ <para>
+ <literal>keepalive_intvl</literal></para>
+ <para>
+ <literal>(2,Wc)</literal></para>
</entry>
<entry>
- <para>Time (in seconds) to repeat unanswered keepalive probes. Setting this value to zero (0) disables keepalives.</para>
+ <para>Time (in seconds) to repeat unanswered keepalive probes. Setting this value to
+ zero (0) disables keepalives.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>keepalive_count</literal></para>
- <para> <literal>(10,Wc)</literal></para>
+ <para>
+ <literal>keepalive_count</literal></para>
+ <para>
+ <literal>(10,Wc)</literal></para>
</entry>
<entry>
- <para>Number of unanswered keepalive probes before pronouncing socket (hence peer) death.</para>
+ <para>Number of unanswered keepalive probes before pronouncing socket (hence peer)
+ death.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>enable_irq_affinity</literal></para>
- <para> <literal>(0,Wc)</literal></para>
+ <para>
+ <literal>enable_irq_affinity</literal></para>
+ <para>
+ <literal>(0,Wc)</literal></para>
</entry>
<entry>
- <para>Boolean that determines whether to enable IRQ affinity. The default is zero (0).</para>
- <para>When set, <literal>socklnd</literal> attempts to maximize performance by handling device interrupts and data movement for particular (hardware) interfaces on particular CPUs. This option is not available on all platforms. This option requires an SMP system to exist and produces best performance with multiple NICs. Systems with multiple CPUs and a single NIC may see increase in the performance with this parameter disabled.</para>
+ <para>Boolean that determines whether to enable IRQ affinity. The default is zero
+ (0).</para>
+ <para>When set, <literal>socklnd</literal> attempts to maximize performance by
+ handling device interrupts and data movement for particular (hardware) interfaces
+ on particular CPUs. This option is not available on all platforms. This option
+ requires an SMP system to exist and produces best performance with multiple NICs.
+ Systems with multiple CPUs and a single NIC may see increase in the performance
+ with this parameter disabled.</para>
</entry>
</row>
<row>
<entry>
- <para> <literal>zc_min_frag</literal></para>
- <para> <literal>(2048,W)</literal></para>
+ <para>
+ <literal>zc_min_frag</literal></para>
+ <para>
+ <literal>(2048,W)</literal></para>
</entry>
<entry>
- <para>Determines the minimum message fragment that should be considered for zero-copy sends. Increasing it above the platform's <literal>PAGE_SIZE </literal>disables all zero copy sends. This option is not available on all platforms.</para>
+ <para>Determines the minimum message fragment that should be considered for
+ zero-copy sends. Increasing it above the platform's <literal>PAGE_SIZE
+ </literal>disables all zero copy sends. This option is not available on all
+ platforms.</para>
</entry>
</row>
</tbody>
<para><literal>n_waitd(1)</literal> sets the number of threads that process completed MX requests (sends and receives).</para>
<para><literal>max_peers(1024)</literal> tells MXLND the upper limit of machines that it will need to communicate with. This affects how many receives it will pre-post and each receive will use one page of memory. Ideally, on clients, this value will be equal to the total number of Lustre servers (MDS and OSS). On servers, it needs to equal the total number of machines in the storage system. cksum (0) turns on small message checksums. It can be used to aid in troubleshooting. MX also provides an optional checksumming feature which can check all messages (large and small). For details, see the MX README.</para>
<para><literal>ntx(256)</literal> is the number of total sends in flight from this machine. In actuality, MXLND reserves half of them for connect messages so make this value twice as large as you want for the total number of sends in flight.</para>
- <para><literal>credits(8)</literal> is the number of in-flight messages for a specific peer. This is part of the flow-control system in Lustre. Increasing this value may improve performance but it requires more memory because each message requires at least one page.</para>
+ <para><literal>credits(8)</literal> is the number of in-flight messages for a specific peer.
+ This is part of the flow-control system in provided by the Lustre software. Increasing this
+ value may improve performance but it requires more memory because each message requires at
+ least one page.</para>
<para><literal>board(0)</literal> is the index of the Myricom NIC. Hosts can have multiple Myricom NICs and this identifies which one MXLND should use. This value must match the board value in your MXLND hosts file for this host.</para>
<para><literal>ep_id(3)</literal> is the MX endpoint ID. Each process that uses MX is required to have at least one MX endpoint to access the MX library and NIC. The ID is a simple index starting at zero (0). This value must match the endpoint ID value in your MXLND hosts file for this host.</para>
<para><literal>polling(0)</literal> determines whether this host will poll or block for MX request completions. A value of 0 blocks and any positive value will poll that many times before blocking. Since polling increases CPU usage, we suggest that you set this to zero (0) on the client and experiment with different values for servers.</para>
git://git.whamcloud.com / doc / manual.git / blobdiff