-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='configurationfilesmoduleparameters'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="configurationfilesmoduleparameters">
<info>
- <title xml:id='configurationfilesmoduleparameters.title'>Configuration Files and Module Parameters</title>
+ <title xml:id="configurationfilesmoduleparameters.title">Configuration Files and Module Parameters</title>
</info>
<para>This section describes configuration files and module parameters and includes the following sections:</para>
- <itemizedlist><listitem>
+ <itemizedlist>
+ <listitem>
<para><xref linkend="dbdoclet.50438293_15350"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438293_78010"/></para>
</listitem>
-
-</itemizedlist>
- <section xml:id="dbdoclet.50438293_15350">
- <title>35.1 Introduction</title>
- <para>LNET network hardware and routing are now configured via module parameters. Parameters should be specified in the /etc/modprobe.conf file, for example:</para>
- <screen>alias lustre llite
-options lnet networks=tcp0,elan0
-</screen>
- <para>The above option specifies that this node should use all the available TCP and Elan interfaces.</para>
- <para>Module parameters are read when the module is first loaded. Type-specific LND modules (for instance, ksocklnd) are loaded automatically by the LNET module when LNET starts (typically upon modprobe ptlrpc).</para>
- <para>Under Linux 2.6, LNET configuration parameters can be viewed under /sys/module/; generic and acceptor parameters under LNET, and LND-specific parameters under the name of the corresponding LND.</para>
- <para>Under Linux 2.4, sysfs is not available, but the LND-specific parameters are accessible via equivalent paths under /proc.</para>
- <para>Important: All old (pre v.1.4.6) Lustre configuration lines should be removed from the module configuration files and replaced with the following. Make sure that CONFIG_KMOD is set in your linux.config so LNET can load the following modules it needs. The basic module files are:</para>
- <para>modprobe.conf (for Linux 2.6)</para>
- <screen>alias lustre llite
-options lnet networks=tcp0,elan0
-</screen>
- <para>modules.conf (for Linux 2.4)</para>
- <screen>alias lustre llite
-options lnet networks=tcp0,elan0
-</screen>
- <para>For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. (Unmarked parameters can only be set when LNET loads for the first time.) Changes to parameters marked with Wc only have effect when connections are established (existing connections are not affected by these changes.)</para>
- </section>
- <section xml:id="dbdoclet.50438293_78010">
- <title>35.2 Module <anchor xml:id="dbdoclet.50438293_marker-1293311" xreflabel=""/>Options</title>
- <itemizedlist><listitem>
- <para> With routed or other multi-network configurations, use ip2nets rather than networks, so all nodes can use the same configuration.</para>
- </listitem>
-
-<listitem>
- <para> For a routed network, use the same 'routes†configuration everywhere. Nodes specified as routers automatically enable forwarding and any routes that are not relevant to a particular node are ignored. Keep a common configuration to guarantee that all nodes have consistent routing tables.</para>
- </listitem>
-
-<listitem>
- <para> A separate modprobe.conf.lnet included from modprobe.conf makes distributing the configuration much easier.</para>
- </listitem>
-
-<listitem>
- <para> If you set config_on_load=1, LNET starts at modprobe time rather than waiting for Lustre to start. This ensures routers start working at module load time.</para>
- </listitem>
-
-</itemizedlist>
- <screen># lctl
-# lctl> net down
-</screen>
- <itemizedlist><listitem>
- <para> Remember the lctl ping {nid} command - it is a handy way to check your LNET configuration.</para>
- </listitem>
-
-</itemizedlist>
- <section remap="h3">
- <title>35.2.1 <anchor xml:id="dbdoclet.50438293_94707" xreflabel=""/>LNET <anchor xml:id="dbdoclet.50438293_marker-1293320" xreflabel=""/>Options</title>
- <para>This section describes LNET options.</para>
- <section remap="h4">
- <title>35.2.1.1 Network Topology</title>
- <para>Network topology module parameters determine which networks a node should join, whether it should route between these networks, and how it communicates with non-local networks.</para>
- <para>Here is a list of various networks and the supported software stacks:</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Network</emphasis></para></entry>
- <entry><para><emphasis role="bold">Software Stack</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> o2ib</para></entry>
- <entry><para> OFED Version 2</para></entry>
- </row>
- <row>
- <entry><para> mx</para></entry>
- <entry><para> Myrinet MX</para></entry>
- </row>
- <row>
- <entry><para> gm</para></entry>
- <entry><para> Myrinet GM-2</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <note><para>Lustre ignores the loopback interface (lo0), but Lustre use any IP addresses aliased to the loopback (by default). When in doubt, explicitly specify networks.</para></note>
- <para><emphasis role="bold">ip2nets</emphasis> ("") is a string that lists globally-available networks, each with a set of IP address ranges. LNET determines the locally-available networks from this list by matching the IP address ranges with the local IPs of a node. The purpose of this option is to be able to use the same modules.conf file across a variety of nodes on different networks. The string has the following syntax.</para>
- <screen><ip2nets> :== <net-match> [ <comment> ] { <net-sep> <net-match> }
-<net-match> :== [ <w> ] <net-spec> <w> <ip-range> { <w> <ip-range> }
-[ <w> ]
-<net-spec> :== <network> [ "(" <interface-list> ")" ]
-<network> :== <nettype> [ <number> ]
-<nettype> :== "tcp" | "elan" | "openib" | ...
-<iface-list> :== <interface> [ "," <iface-list> ]
-<ip-range> :== <r-expr> "." <r-expr> "." <r-expr> "." <r-expr>
-<r-expr> :== <number> | "*" | "[" <r-list> "]"
-<r-list> :== <range> [ "," <r-list> ]
-<range> :== <number> [ "-" <number> [ "/" <number> ] ]
-<comment :== "#" { <non-net-sep-chars> }
-<net-sep> :== ";" | "\n"
-<w> :== <whitespace-chars> { <whitespace-chars> }
-</screen>
- <para><<emphasis role="bold">net-spec</emphasis>> contains enough information to uniquely identify the network and load an appropriate LND. The LND determines the missing "address-within-network" part of the NID based on the interfaces it can use.</para>
- <para><<emphasis role="bold">iface-list</emphasis>> specifies which hardware interface the network can use. If omitted, all interfaces are used. LNDs that do not support the <iface-list> syntax cannot be configured to use particular interfaces and just use what is there. Only a single instance of these LNDs can exist on a node at any time, and <iface-list> must be omitted.</para>
- <para><<emphasis role="bold">net-match</emphasis>> entries are scanned in the order declared to see if one of the node's IP addresses matches one of the <ip-range> expressions. If there is a match, <net-spec> specifies the network to instantiate. Note that it is the first match for a particular network that counts. This can be used to simplify the match expression for the general case by placing it after the special cases. For example:</para>
- <screen>ip2nets="tcp(eth1,eth2) 134.32.1.[4-10/2]; tcp(eth1) *.*.*.*"
-</screen>
- <para>4 nodes on the 134.32.1.* network have 2 interfaces (134.32.1.{4,6,8,10}) but all the rest have 1.</para>
- <screen>ip2nets="<emphasis role="bold">vib</emphasis> 192.168.0.*; tcp(eth2) 192.168.0.[1,7,4,12]"
-</screen>
- <para>This describes an IB cluster on 192.168.0.*. Four of these nodes also have IP interfaces; these four could be used as routers.</para>
- <para>Note that match-all expressions (For instance, *.*.*.*) effectively mask all other</para>
- <para> <net-match> entries specified after them. They should be used with caution.</para>
- <para>Here is a more complicated situation, the route parameter is explained below. We have:</para>
- <itemizedlist><listitem>
- <para> Two TCP subnets</para>
- </listitem>
-
-<listitem>
- <para> One Elan subnet</para>
- </listitem>
-
-<listitem>
- <para> One machine set up as a router, with both TCP and Elan interfaces</para>
- </listitem>
-
-<listitem>
- <para> IP over Elan configured, but only IP will be used to label the nodes.</para>
- </listitem>
-
-</itemizedlist>
- <screen>options lnet ip2nets=â€tcp 198.129.135.* 192.128.88.98; \
- elan 198.128.88.98 198.129.135.3; \
- routes='cp 1022@elan # Elan NID of router; \
- elan 198.128.88.98@tcp # TCP NID of router '
-</screen>
- </section>
- <section remap="h4">
- <title>35.2.1.2 networks ("tcp")</title>
- <para>This is an alternative to "ip2nets" which can be used to specify the networks to be instantiated explicitly. The syntax is a simple comma separated list of <net-spec>s (see above). The default is only used if neither 'ip2nets†nor 'networks†is specified.</para>
- </section>
- <section remap="h4">
- <title>35.2.1.3 routes ("")</title>
- <para>This is a string that lists networks and the NIDs of routers that forward to them.</para>
- <para>It has the following syntax (<w> is one or more whitespace characters):</para>
- <screen><routes> :== <route>{ ; <route> }
-<route> :== [<net>[<w><hopcount>]<w><nid>{<w><nid>}
-</screen>
- <para>So a node on the network tcp1 that needs to go through a router to get to the Elan network:</para>
- <screen>options lnet networks=tcp1 routes="elan 1 192.168.2.2@tcpA"
-</screen>
- <para>The hopcount is used to help choose the best path between multiply-routed configurations.</para>
- <para>A simple but powerful expansion syntax is provided, both for target networks and router NIDs as follows.</para>
- <screen><expansion> :== "[" <entry> { "," <entry> } "]"
-<entry> :== <numeric range> | <non-numeric item>
-<numeric range> :== <number> [ "-" <number> [ "/" <number> ] ]
-</screen>
- <para>The expansion is a list enclosed in square brackets. Numeric items in the list may be a single number, a contiguous range of numbers, or a strided range of numbers. For example, routes="elan 192.168.1.[22-24]@tcp" says that network elan0 is adjacent (hopcount defaults to 1); and is accessible via 3 routers on the tcp0 network (192.168.1.22@tcp, 192.168.1.23@tcp and 192.168.1.24@tcp).</para>
- <para>routes="[tcp,<emphasis role="bold">vib</emphasis>] 2 [8-14/2]@elan" says that 2 networks (tcp0 and vib0) are accessible through 4 routers (8@elan, 10@elan, 12@elan and 14@elan). The hopcount of 2 means that traffic to both these networks will be traversed 2 routers - first one of the routers specified in this entry, then one more.</para>
- <para>Duplicate entries, entries that route to a local network, and entries that specify routers on a non-local network are ignored.</para>
- <para>Equivalent entries are resolved in favor of the route with the shorter hopcount. The hopcount, if omitted, defaults to 1 (the remote network is adjacent).</para>
- <para>It is an error to specify routes to the same destination with routers on different local networks.</para>
- <para>If the target network string contains no expansions, then the hopcount defaults to 1 and may be omitted (that is, the remote network is adjacent). In practice, this is true for most multi-network configurations. It is an error to specify an inconsistent hop count for a given target network. This is why an explicit hopcount is required if the target network string specifies more than one network.</para>
- </section>
- <section remap="h4">
- <title>35.2.1.4 forwarding ("")</title>
- <para>This is a string that can be set either to "enabled" or "disabled" for explicit control of whether this node should act as a router, forwarding communications between all local networks.</para>
- <para>A standalone router can be started by simply starting LNET ('modprobe ptlrpcâ€) with appropriate network topology options.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Variable</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> acceptor</para></entry>
- <entry><para> The acceptor is a TCP/IP service that some LNDs use to establish communications. If a local network requires it and it has not been disabled, the acceptor listens on a single port for connection requests that it redirects to the appropriate local network. The acceptor is part of the LNET module and configured by the following options:</para><itemizedlist><listitem>
- <para><emphasis role="bold">secure</emphasis> - Accept connections only from reserved TCP ports (< 1023).</para>
- </listitem>
-<listitem>
- <para><emphasis role="bold">all</emphasis> - Accept connections from any TCP port. NOTE: this is required for liblustre clients to allow connections on non-privileged ports.</para>
- </listitem>
-<listitem>
- <para><emphasis role="bold">none</emphasis> - Do not run the acceptor.</para>
- </listitem>
-</itemizedlist></entry>
- </row>
- <row>
- <entry><para> accept_port</para><para> (988)</para></entry>
- <entry><para> Port number on which the acceptor should listen for connection requests. All nodes in a site configuration that require an acceptor must use the same port.</para></entry>
- </row>
- <row>
- <entry><para> accept_backlog</para><para> (127)</para></entry>
- <entry><para> Maximum length that the queue of pending connections may grow to (see listen(2)).</para></entry>
- </row>
- <row>
- <entry><para> accept_timeout</para><para> (5, W)</para></entry>
- <entry><para> Maximum time in seconds the acceptor is allowed to block while communicating with a peer.</para></entry>
- </row>
- <row>
- <entry><para> accept_proto_version</para></entry>
- <entry><para> Version of the acceptor protocol that should be used by outgoing connection requests. It defaults to the most recent acceptor protocol version, but it may be set to the previous version to allow the node to initiate connections with nodes that only understand that version of the acceptor protocol. The acceptor can, with some restrictions, handle either version (that is, it can accept connections from both 'old' and 'new' peers). For the current version of the acceptor protocol (version 1), the acceptor is compatible with old peers if it is only required by a single local network.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <para></para>
- <para></para>
- </section>
- </section>
- <section remap="h3">
- <title>35.2.2 SOCKLND <anchor xml:id="dbdoclet.50438293_marker-1293448" xreflabel=""/>Kernel TCP/IP LND</title>
- <para>The SOCKLND kernel TCP/IP LND (socklnd) is connection-based and uses the acceptor to establish communications via sockets with its peers.</para>
- <para>It supports multiple instances and load balances dynamically over multiple interfaces. If no interfaces are specified by the ip2nets or networks module parameter, all non-loopback IP interfaces are used. The address-within-network is determined by the address of the first IP interface an instance of the socklnd encounters.</para>
- <para>Consider a node on the 'edge†of an InfiniBand network, with a low-bandwidth management Ethernet (eth0), IP over IB configured (ipoib0), and a pair of GigE NICs (eth1,eth2) providing off-cluster connectivity. This node should be configured with "networks=<emphasis role="bold">vib</emphasis>,tcp(eth1,eth2)†to ensure that the socklnd ignores the management Ethernet and IPoIB.</para>
+ </itemizedlist>
+ <section xml:id="dbdoclet.50438293_15350">
+ <title>35.1 Introduction</title>
+ <para>LNET network hardware and routing are now configured via module parameters. Parameters should be specified in the <literal>/etc/modprobe.conf </literal>file, for example:</para>
+ <screen>alias lustre llite
+options lnet networks=tcp0,elan0</screen>
+ <para>The above option specifies that this node should use all the available TCP and Elan interfaces.</para>
+ <para>Module parameters are read when the module is first loaded. Type-specific LND modules (for instance, <literal>ksocklnd</literal>) are loaded automatically by the LNET module when LNET starts (typically upon <literal>modprobe ptlrpc</literal>).</para>
+ <para>Under Linux 2.6, LNET configuration parameters can be viewed under <literal>/sys/module/</literal>; generic and acceptor parameters under LNET, and LND-specific parameters under the name of the corresponding LND.</para>
+ <para>Under Linux 2.4, <literal>sysfs</literal> is not available, but the LND-specific parameters are accessible via equivalent paths under <literal>/proc</literal>.</para>
+ <para>Important: All old (pre v.1.4.6) Lustre configuration lines should be removed from the module configuration files and replaced with the following. Make sure that <literal>CONFIG_KMOD</literal> is set in your <literal>linux.config</literal> so LNET can load the following modules it needs. The basic module files are:</para>
+ <para><literal>modprobe.conf </literal>(for Linux 2.6)</para>
+ <screen>alias lustre llite
+options lnet networks=tcp0,elan0</screen>
+ <para><literal>modules.conf</literal> (for Linux 2.4)</para>
+ <screen>alias lustre llite
+options lnet networks=tcp0,elan0</screen>
+ <para>For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. (Unmarked parameters can only be set when LNET loads for the first time.) Changes to parameters marked with <literal>Wc</literal> only have effect when connections are established (existing connections are not affected by these changes.)</para>
+ </section>
+ <section xml:id="dbdoclet.50438293_78010">
+ <title>35.2 Module Options</title>
+ <itemizedlist>
+ <listitem>
+ <para>With routed or other multi-network configurations, use <literal>ip2nets</literal> rather than networks, so all nodes can use the same configuration.</para>
+ </listitem>
+ <listitem>
+ <para>For a routed network, use the same 'routes' configuration everywhere. Nodes specified as routers automatically enable forwarding and any routes that are not relevant to a particular node are ignored. Keep a common configuration to guarantee that all nodes have consistent routing tables.</para>
+ </listitem>
+ <listitem>
+ <para>A separate <literal>modprobe.conf.lnet</literal> included from <literal>modprobe.conf</literal> makes distributing the configuration much easier.</para>
+ </listitem>
+ <listitem>
+ <para>If you set <literal>config_on_load=1</literal>, LNET starts at <literal>modprobe</literal> time rather than waiting for Lustre to start. This ensures routers start working at module load time.</para>
+ </listitem>
+ </itemizedlist>
+ <screen># lctl
+# lctl> net down</screen>
+ <itemizedlist>
+ <listitem>
+ <para>Remember the <literal>lctl ping {nid}</literal> command - it is a handy way to check your LNET configuration.</para>
+ </listitem>
+ </itemizedlist>
+ <section remap="h3">
+ <title>35.2.1 LNET Options</title>
+ <para>This section describes LNET options.</para>
+ <section remap="h4">
+ <title>35.2.1.1 Network Topology</title>
+ <para>Network topology module parameters determine which networks a node should join, whether it should route between these networks, and how it communicates with non-local networks.</para>
+ <para>Here is a list of various networks and the supported software stacks:</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold">Variable</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
+ <entry>
+ <para><emphasis role="bold">Network</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Software Stack</emphasis></para>
+ </entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> timeout</para><para> (50,W)</para></entry>
- <entry><para> Time (in seconds) that communications may be stalled before the LND completes them with failure.</para></entry>
- </row>
- <row>
- <entry><para> nconnds</para><para> (4)</para></entry>
- <entry><para> Sets the number of connection daemons.</para></entry>
+ <entry>
+ <para> o2ib</para>
+ </entry>
+ <entry>
+ <para> OFED Version 2</para>
+ </entry>
</row>
<row>
- <entry><para> min_reconnectms</para><para> (1000,W)</para></entry>
- <entry><para> Minimum connection retry interval (in milliseconds). After a failed connection attempt, this is the time that must elapse before the first retry. As connections attempts fail, this time is doubled on each successive retry up to a maximum of 'max_reconnectms'.</para></entry>
+ <entry>
+ <para> mx</para>
+ </entry>
+ <entry>
+ <para> Myrinet MX</para>
+ </entry>
</row>
<row>
- <entry><para> max_reconnectms</para><para> (6000,W)</para></entry>
- <entry><para> Maximum connection retry interval (in milliseconds).</para></entry>
- </row>
- <row>
- <entry><para> eager_ack</para><para> (0 on linux,</para><para> 1 on darwin,W)</para></entry>
- <entry><para> Boolean that determines whether the socklnd should attempt to flush sends on message boundaries.</para></entry>
- </row>
- <row>
- <entry><para> typed_conns</para><para> (1,Wc)</para></entry>
- <entry><para> Boolean that determines whether the socklnd should use different sockets for different types of messages. When clear, all communication with a particular peer takes place on the same socket. Otherwise, separate sockets are used for bulk sends, bulk receives and everything else.</para></entry>
- </row>
- <row>
- <entry><para> min_bulk</para><para> (1024,W)</para></entry>
- <entry><para> Determines when a message is considered "bulk".</para></entry>
- </row>
- <row>
- <entry><para> tx_buffer_size, rx_buffer_size</para><para> (8388608,Wc)</para></entry>
- <entry><para> Socket buffer sizes. Setting this option to zero (0), allows the system to auto-tune buffer sizes. WARNING: Be very careful changing this value as improper sizing can harm performance.</para></entry>
- </row>
- <row>
- <entry><para> nagle</para><para> (0,Wc)</para></entry>
- <entry><para> Boolean that determines if nagle should be enabled. It should never be set in production systems.</para></entry>
- </row>
- <row>
- <entry><para> keepalive_idle</para><para> (30,Wc)</para></entry>
- <entry><para> Time (in seconds) that a socket can remain idle before a keepalive probe is sent. Setting this value to zero (0) disables keepalives.</para></entry>
- </row>
- <row>
- <entry><para> keepalive_intvl</para><para> (2,Wc)</para></entry>
- <entry><para> Time (in seconds) to repeat unanswered keepalive probes. Setting this value to zero (0) disables keepalives.</para></entry>
- </row>
- <row>
- <entry><para> keepalive_count</para><para> (10,Wc)</para></entry>
- <entry><para> Number of unanswered keepalive probes before pronouncing socket (hence peer) death.</para></entry>
- </row>
- <row>
- <entry><para> enable_irq_affinity</para><para> (0,Wc)</para></entry>
- <entry><para> Boolean that determines whether to enable IRQ affinity. The default is zero (0).</para><para>When set, socklnd attempts to maximize performance by handling device interrupts and data movement for particular (hardware) interfaces on particular CPUs. This option is not available on all platforms. This option requires an SMP system to exist and produces best performance with multiple NICs. Systems with multiple CPUs and a single NIC may see increase in the performance with this parameter disabled.</para></entry>
- </row>
- <row>
- <entry><para> zc_min_frag</para><para> (2048,W)</para></entry>
- <entry><para> Determines the minimum message fragment that should be considered for zero-copy sends. Increasing it above the platform's PAGE_SIZE disables all zero copy sends. This option is not available on all platforms.</para></entry>
+ <entry>
+ <para> gm</para>
+ </entry>
+ <entry>
+ <para> Myrinet GM-2</para>
+ </entry>
</row>
</tbody>
</tgroup>
</informaltable>
+ <note>
+ <para>Lustre ignores the loopback interface (<literal>lo0</literal>), but Lustre use any IP addresses aliased to the loopback (by default). When in doubt, explicitly specify networks.</para>
+ </note>
+ <para><literal>ip2nets</literal> ("") is a string that lists globally-available networks, each with a set of IP address ranges. LNET determines the locally-available networks from this list by matching the IP address ranges with the local IPs of a node. The purpose of this option is to be able to use the same <literal>modules.conf</literal> file across a variety of nodes on different networks. The string has the following syntax.</para>
+ <screen><ip2nets> :== <net-match> [ <comment> ] { <net-sep> <net-match> }
+<net-match> :== [ <w> ] <net-spec> <w> <ip-range> { <w> <ip-range> }
+[ <w> ]
+<net-spec> :== <network> [ "(" <interface-list> ")" ]
+<network> :== <nettype> [ <number> ]
+<nettype> :== "tcp" | "elan" | "openib" | ...
+<iface-list> :== <interface> [ "," <iface-list> ]
+<ip-range> :== <r-expr> "." <r-expr> "." <r-expr> "." <r-expr>
+<r-expr> :== <number> | "*" | "[" <r-list> "]"
+<r-list> :== <range> [ "," <r-list> ]
+<range> :== <number> [ "-" <number> [ "/" <number> ] ]
+<comment :== "#" { <non-net-sep-chars> }
+<net-sep> :== ";" | "\n"
+<w> :== <whitespace-chars> { <whitespace-chars> }
+</screen>
+ <para><literal><net-spec></literal> contains enough information to uniquely identify the network and load an appropriate LND. The LND determines the missing "address-within-network" part of the NID based on the interfaces it can use.</para>
+ <para><literal><iface-list></literal> specifies which hardware interface the network can use. If omitted, all interfaces are used. LNDs that do not support the <literal><iface-list></literal> syntax cannot be configured to use particular interfaces and just use what is there. Only a single instance of these LNDs can exist on a node at any time, and <literal><iface-list></literal> must be omitted.</para>
+ <para><literal><net-match></literal> entries are scanned in the order declared to see if one of the node's IP addresses matches one of the <literal><ip-range></literal> expressions. If there is a match, <literal><net-spec></literal> specifies the network to instantiate. Note that it is the first match for a particular network that counts. This can be used to simplify the match expression for the general case by placing it after the special cases. For example:</para>
+ <screen>ip2nets="tcp(eth1,eth2) 134.32.1.[4-10/2]; tcp(eth1) *.*.*.*"</screen>
+ <para>4 nodes on the 134.32.1.* network have 2 interfaces (134.32.1.{4,6,8,10}) but all the rest have 1.</para>
+ <screen>ip2nets="<emphasis role="bold">vib</emphasis> 192.168.0.*; tcp(eth2) 192.168.0.[1,7,4,12]" </screen>
+ <para>This describes an IB cluster on 192.168.0.*. Four of these nodes also have IP interfaces; these four could be used as routers.</para>
+ <para>Note that match-all expressions (For instance, <literal>*.*.*.*</literal>) effectively mask all other</para>
+ <para> <literal><net-match></literal> entries specified after them. They should be used with caution.</para>
+ <para>Here is a more complicated situation, the route parameter is explained below. We have:</para>
+ <itemizedlist>
+ <listitem>
+ <para>Two TCP subnets</para>
+ </listitem>
+ <listitem>
+ <para>One Elan subnet</para>
+ </listitem>
+ <listitem>
+ <para>One machine set up as a router, with both TCP and Elan interfaces</para>
+ </listitem>
+ <listitem>
+ <para>IP over Elan configured, but only IP will be used to label the nodes.</para>
+ </listitem>
+ </itemizedlist>
+ <screen>options lnet ip2nets=â€tcp 198.129.135.* 192.128.88.98; \
+ elan 198.128.88.98 198.129.135.3; \
+ routes='cp 1022@elan # Elan NID of router; \
+ elan 198.128.88.98@tcp # TCP NID of router '</screen>
</section>
- <section remap="h3">
- <title>35.2.3 Portals <anchor xml:id="dbdoclet.50438293_marker-1293719" xreflabel=""/>LND (Linux)</title>
- <para>The Portals LND Linux (ptllnd) can be used as a interface layer to communicate with Sandia Portals networking devices. This version is intended to work on Cray XT3 Linux nodes that use Cray Portals as a network transport.</para>
- <para><emphasis role="bold">Message Buffers</emphasis></para>
- <para>When ptllnd starts up, it allocates and posts sufficient message buffers to allow all expected peers (set by concurrent_peers) to send one unsolicited message. The first message that a peer actually sends is a</para>
- <para>(so-called) "HELLO" message, used to negotiate how much additional buffering to setup (typically 8 messages). If 10000 peers actually exist, then enough buffers are posted for 80000 messages.</para>
- <para>The maximum message size is set by the max_msg_size module parameter (default value is 512). This parameter sets the bulk transfer breakpoint. Below this breakpoint, payload data is sent in the message itself. Above this breakpoint, a buffer descriptor is sent and the receiver gets the actual payload.</para>
- <para>The buffer size is set by the rxb_npages module parameter (default value is 1). The default conservatively avoids allocation problems due to kernel memory fragmentation. However, increasing this value to 2 is probably not risky.</para>
- <para>The ptllnd also keeps an additional rxb_nspare buffers (default value is 8) posted to account for full buffers being handled.</para>
- <para>Assuming a 4K page size with 10000 peers, 1258 buffers can be expected to be posted at startup, increasing to a maximum of 10008 as peers that are actually connected. By doubling rxb_npages halving max_msg_size, this number can be reduced by a factor of 4.</para>
- <para><emphasis role="bold">ME/MD Queue Length</emphasis></para>
- <para>The ptllnd uses a single portal set by the portal module parameter (default value of 9) for both message and bulk buffers. Message buffers are always attached with PTL_INS_AFTER and match anything sent with "message" matchbits. Bulk buffers are always attached with PTL_INS_BEFORE and match only specific matchbits for that particular bulk transfer.</para>
- <para>This scheme assumes that the majority of ME / MDs posted are for "message" buffers, and that the overhead of searching through the preceding "bulk" buffers is acceptable. Since the number of "bulk" buffers posted at any time is also dependent on the bulk transfer breakpoint set by max_msg_size, this seems like an issue worth measuring at scale.</para>
- <para><emphasis role="bold">TX Descriptors</emphasis></para>
- <para>The ptllnd has a pool of so-called "tx descriptors", which it uses not only for outgoing messages, but also to hold state for bulk transfers requested by incoming messages. This pool should scale with the total number of peers.</para>
- <para>To enable the building of the Portals LND (ptllnd.ko) configure with this option:</para>
- <screen>./configure --with-portals=<path-to-portals-headers></screen>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Variable</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> ntx</para><para> (256)</para></entry>
- <entry><para> Total number of messaging descriptors.</para></entry>
- </row>
- <row>
- <entry><para> concurrent_peers</para><para> (1152)</para></entry>
- <entry><para> Maximum number of concurrent peers. Peers that attempt to connect beyond the maximum are not allowed.</para></entry>
- </row>
- <row>
- <entry><para> peer_hash_table_size</para><para> (101)</para></entry>
- <entry><para> Number of hash table slots for the peers. This number should scale with concurrent_peers. The size of the peer hash table is set by the module parameter peer_hash_table_size which defaults to a value of 101. This number should be prime to ensure the peer hash table is populated evenly. It is advisable to increase this value to 1001 for ~10000 peers.</para></entry>
- </row>
- <row>
- <entry><para> cksum</para><para> (0)</para></entry>
- <entry><para> Set to non-zero to enable message (not RDMA) checksums for outgoing packets. Incoming packets are always check-summed if necessary, independent of this value.</para></entry>
- </row>
- <row>
- <entry><para> timeout</para><para> (50)</para></entry>
- <entry><para> Amount of time (in seconds) that a request can linger in a peers-active queue before the peer is considered dead.</para></entry>
- </row>
- <row>
- <entry><para> portal</para><para> (9)</para></entry>
- <entry><para> Portal ID to use for the ptllnd traffic.</para></entry>
- </row>
- <row>
- <entry><para> rxb_npages</para><para> (64 * #cpus)</para></entry>
- <entry><para> Number of pages in an RX buffer.</para></entry>
- </row>
- <row>
- <entry><para> credits</para><para> (128)</para></entry>
- <entry><para> Maximum total number of concurrent sends that are outstanding to a single peer at a given time.</para></entry>
- </row>
- <row>
- <entry><para> peercredits</para><para> (8)</para></entry>
- <entry><para> Maximum number of concurrent sends that are outstanding to a single peer at a given time.</para></entry>
- </row>
- <row>
- <entry><para> max_msg_size</para><para> (512)</para></entry>
- <entry><para> Maximum immediate message size. This MUST be the same on all nodes in a cluster. A peer that connects with a different max_msg_size value will be rejected.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
+ <section remap="h4">
+ <title>35.2.1.2 networks ("tcp")</title>
+ <para>This is an alternative to "<literal>ip2nets</literal>" which can be used to specify the networks to be instantiated explicitly. The syntax is a simple comma separated list of <literal><net-spec></literal>s (see above). The default is only used if neither 'ip2nets' nor 'networks' is specified.</para>
</section>
- <section remap="h3">
- <title>35.2.4 MX <anchor xml:id="dbdoclet.50438293_marker-1295997" xreflabel=""/>LND</title>
- <para>MXLND supports a number of load-time parameters using Linux's module parameter system. The following variables are available:</para>
+ <section remap="h4">
+ <title>35.2.1.3 routes ("")</title>
+ <para>This is a string that lists networks and the NIDs of routers that forward to them.</para>
+ <para>It has the following syntax (<literal><w></literal> is one or more whitespace characters):</para>
+ <screen><routes> :== <route>{ ; <route> }
+<route> :== [<net>[<w><hopcount>]<w><nid>{<w><nid>}</screen>
+ <para>So a node on the network <literal>tcp1</literal> that needs to go through a router to get to the Elan network:</para>
+ <screen>options lnet networks=tcp1 routes="elan 1 192.168.2.2@tcpA"</screen>
+ <para>The hopcount is used to help choose the best path between multiply-routed configurations.</para>
+ <para>A simple but powerful expansion syntax is provided, both for target networks and router NIDs as follows.</para>
+ <screen><expansion> :== "[" <entry> { "," <entry> } "]"
+<entry> :== <numeric range> | <non-numeric item>
+<numeric range> :== <number> [ "-" <number> [ "/" <number> ] ]</screen>
+ <para>The expansion is a list enclosed in square brackets. Numeric items in the list may be a single number, a contiguous range of numbers, or a strided range of numbers. For example, <literal>routes="elan 192.168.1.[22-24]@tcp"</literal> says that network <literal>elan0</literal> is adjacent (hopcount defaults to 1); and is accessible via 3 routers on the <literal>tcp0</literal> network (<literal>192.168.1.22@tcp</literal>, <literal>192.168.1.23@tcp</literal> and <literal>192.168.1.24@tcp</literal>).</para>
+ <para><literal>routes="[tcp,vib] 2 [8-14/2]@elan"</literal> says that 2 networks (<literal>tcp0</literal> and <literal>vib0</literal>) are accessible through 4 routers (<literal>8@elan</literal>, <literal>10@elan</literal>, <literal>12@elan</literal> and <literal>14@elan</literal>). The hopcount of 2 means that traffic to both these networks will be traversed 2 routers - first one of the routers specified in this entry, then one more.</para>
+ <para>Duplicate entries, entries that route to a local network, and entries that specify routers on a non-local network are ignored.</para>
+ <para>Equivalent entries are resolved in favor of the route with the shorter hopcount. The hopcount, if omitted, defaults to 1 (the remote network is adjacent).</para>
+ <para>It is an error to specify routes to the same destination with routers on different local networks.</para>
+ <para>If the target network string contains no expansions, then the hopcount defaults to 1 and may be omitted (that is, the remote network is adjacent). In practice, this is true for most multi-network configurations. It is an error to specify an inconsistent hop count for a given target network. This is why an explicit hopcount is required if the target network string specifies more than one network.</para>
+ </section>
+ <section remap="h4">
+ <title>35.2.1.4 forwarding ("")</title>
+ <para>This is a string that can be set either to "<literal>enabled</literal>" or "<literal>disabled</literal>" for explicit control of whether this node should act as a router, forwarding communications between all local networks.</para>
+ <para>A standalone router can be started by simply starting LNET ('<literal>modprobe ptlrpc</literal>') with appropriate network topology options.</para>
<informaltable frame="all">
<tgroup cols="2">
<colspec colname="c1" colwidth="50*"/>
<colspec colname="c2" colwidth="50*"/>
<thead>
<row>
- <entry><para><emphasis role="bold">Variable</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
+ <entry>
+ <para><emphasis role="bold">Variable</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
</row>
</thead>
<tbody>
<row>
- <entry><para> n_waitd</para></entry>
- <entry><para> Number of completion daemons.</para></entry>
- </row>
- <row>
- <entry><para> max_peers</para></entry>
- <entry><para> Maximum number of peers that may connect.</para></entry>
- </row>
- <row>
- <entry><para> cksum</para></entry>
- <entry><para> Enables small message (< 4 KB) checksums if set to a non-zero value.</para></entry>
- </row>
- <row>
- <entry><para> ntx</para></entry>
- <entry><para> Number of total tx message descriptors.</para></entry>
- </row>
- <row>
- <entry><para> credits</para></entry>
- <entry><para> Number of concurrent sends to a single peer.</para></entry>
- </row>
- <row>
- <entry><para> board</para></entry>
- <entry><para> Index value of the Myrinet board (NIC).</para></entry>
- </row>
- <row>
- <entry><para> ep_id</para></entry>
- <entry><para> MX endpoint ID.</para></entry>
- </row>
- <row>
- <entry><para> polling</para></entry>
- <entry><para> Use zero (0) to block (wait). A value > 0 will poll that many times before blocking.</para></entry>
- </row>
- <row>
- <entry><para> hosts</para></entry>
- <entry><para> IP-to-hostname resolution file.</para></entry>
+ <entry>
+ <para> <literal>acceptor</literal></para>
+ </entry>
+ <entry>
+ <para>The acceptor is a TCP/IP service that some LNDs use to establish communications. If a local network requires it and it has not been disabled, the acceptor listens on a single port for connection requests that it redirects to the appropriate local network. The acceptor is part of the LNET module and configured by the following options:</para>
+ <itemizedlist>
+ <listitem>
+ <para><literal>secure</literal> - Accept connections only from reserved TCP ports (< 1023).</para>
+ </listitem>
+ <listitem>
+ <para><literal>all</literal> - Accept connections from any TCP port. </para>
+ <note>
+ <para>This is required for liblustre clients to allow connections on non-privileged ports.</para>
+ </note>
+ </listitem>
+ <listitem>
+ <para><literal>none</literal> - Do not run the acceptor.</para>
+ </listitem>
+ </itemizedlist>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>accept_port</literal></para>
+ <para> <literal>(988)</literal></para>
+ </entry>
+ <entry>
+ <para> Port number on which the acceptor should listen for connection requests. All nodes in a site configuration that require an acceptor must use the same port.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>accept_backlog</literal></para>
+ <para> <literal>(127)</literal></para>
+ </entry>
+ <entry>
+ <para>Maximum length that the queue of pending connections may grow to (see listen(2)).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>accept_timeout</literal></para>
+ <para> <literal>(5, W)</literal></para>
+ </entry>
+ <entry>
+ <para>Maximum time in seconds the acceptor is allowed to block while communicating with a peer.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>accept_proto_version</literal></para>
+ </entry>
+ <entry>
+ <para>Version of the acceptor protocol that should be used by outgoing connection requests. It defaults to the most recent acceptor protocol version, but it may be set to the previous version to allow the node to initiate connections with nodes that only understand that version of the acceptor protocol. The acceptor can, with some restrictions, handle either version (that is, it can accept connections from both 'old' and 'new' peers). For the current version of the acceptor protocol (version 1), the acceptor is compatible with old peers if it is only required by a single local network.</para>
+ </entry>
</row>
</tbody>
</tgroup>
</informaltable>
- <para>Of the described variables, only hosts is required. It must be the absolute path to the MXLND hosts file.</para>
- <para>For example:</para>
- <screen>options kmxlnd hosts=/etc/hosts.mxlnd
-</screen>
- <para>The file format for the hosts file is:</para>
- <screen>IP HOST BOARD EP_ID
-</screen>
- <para>The values must be space and/or tab separated where:</para>
- <para>IP is a valid IPv4 address</para>
- <para>HOST is the name returned by `hostname` on that machine</para>
- <para>BOARD is the index of the Myricom NIC (0 for the first card, etc.)</para>
- <para>EP_ID is the MX endpoint ID</para>
- <para>To obtain the optimal performance for your platform, you may want to vary the remaining options.</para>
- <para>n_waitd (1) sets the number of threads that process completed MX requests (sends and receives).</para>
- <para>max_peers (1024) tells MXLND the upper limit of machines that it will need to communicate with. This affects how many receives it will pre-post and each receive will use one page of memory. Ideally, on clients, this value will be equal to the total number of Lustre servers (MDS and OSS). On servers, it needs to equal the total number of machines in the storage system. cksum (0) turns on small message checksums. It can be used to aid in troubleshooting. MX also provides an optional checksumming feature which can check all messages (large and small). For details, see the MX README.</para>
- <para>ntx (256) is the number of total sends in flight from this machine. In actuality, MXLND reserves half of them for connect messages so make this value twice as large as you want for the total number of sends in flight.</para>
- <para>credits (8) is the number of in-flight messages for a specific peer. This is part of the flow-control system in Lustre. Increasing this value may improve performance but it requires more memory because each message requires at least one page.</para>
- <para>board (0) is the index of the Myricom NIC. Hosts can have multiple Myricom NICs and this identifies which one MXLND should use. This value must match the board value in your MXLND hosts file for this host.</para>
- <para>ep_id (3) is the MX endpoint ID. Each process that uses MX is required to have at least one MX endpoint to access the MX library and NIC. The ID is a simple index starting at zero (0). This value must match the endpoint ID value in your MXLND hosts file for this host.</para>
- <para>polling (0) determines whether this host will poll or block for MX request completions. A value of 0 blocks and any positive value will poll that many times before blocking. Since polling increases CPU usage, we suggest that you set this to zero (0) on the client and experiment with different values for servers.</para>
</section>
</section>
+ <section remap="h3">
+ <title>35.2.2 <literal>SOCKLND</literal> Kernel TCP/IP LND</title>
+ <para>The <literal>SOCKLND</literal> kernel TCP/IP LND (<literal>socklnd</literal>) is connection-based and uses the acceptor to establish communications via sockets with its peers.</para>
+ <para>It supports multiple instances and load balances dynamically over multiple interfaces. If no interfaces are specified by the <literal>ip2nets</literal> or networks module parameter, all non-loopback IP interfaces are used. The address-within-network is determined by the address of the first IP interface an instance of the <literal>socklnd</literal> encounters.</para>
+ <para>Consider a node on the 'edge' of an InfiniBand network, with a low-bandwidth management Ethernet (<literal>eth0</literal>), IP over IB configured (<literal>ipoib0</literal>), and a pair of GigE NICs (<literal>eth1</literal>,<literal>eth2</literal>) providing off-cluster connectivity. This node should be configured with '<literal>networks=vib,tcp(eth1,eth2)</literal>' to ensure that the <literal>socklnd</literal> ignores the management Ethernet and IPoIB.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Variable</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>timeout</literal></para>
+ <para> <literal>(50,W)</literal></para>
+ </entry>
+ <entry>
+ <para>Time (in seconds) that communications may be stalled before the LND completes them with failure.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>nconnds</literal></para>
+ <para> <literal>(4)</literal></para>
+ </entry>
+ <entry>
+ <para>Sets the number of connection daemons.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>min_reconnectms</literal></para>
+ <para> <literal>(1000,W)</literal></para>
+ </entry>
+ <entry>
+ <para>Minimum connection retry interval (in milliseconds). After a failed connection attempt, this is the time that must elapse before the first retry. As connections attempts fail, this time is doubled on each successive retry up to a maximum of '<literal>max_reconnectms</literal>'.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>max_reconnectms</literal></para>
+ <para> <literal>(6000,W)</literal></para>
+ </entry>
+ <entry>
+ <para>Maximum connection retry interval (in milliseconds).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>eager_ack</literal></para>
+ <para> <literal>(0 on linux,</literal></para>
+ <para> <literal>1 on darwin,W)</literal></para>
+ </entry>
+ <entry>
+ <para>Boolean that determines whether the <literal>socklnd</literal> should attempt to flush sends on message boundaries.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>typed_conns</literal></para>
+ <para> <literal>(1,Wc)</literal></para>
+ </entry>
+ <entry>
+ <para>Boolean that determines whether the <literal>socklnd</literal> should use different sockets for different types of messages. When clear, all communication with a particular peer takes place on the same socket. Otherwise, separate sockets are used for bulk sends, bulk receives and everything else.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>min_bulk</literal></para>
+ <para> <literal>(1024,W)</literal></para>
+ </entry>
+ <entry>
+ <para>Determines when a message is considered "bulk".</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>tx_buffer_size, rx_buffer_size</literal></para>
+ <para> <literal>(8388608,Wc)</literal></para>
+ </entry>
+ <entry>
+ <para>Socket buffer sizes. Setting this option to zero (0), allows the system to auto-tune buffer sizes. </para>
+ <warning>
+ <para>Be very careful changing this value as improper sizing can harm performance.</para>
+ </warning>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>nagle</literal></para>
+ <para> <literal>(0,Wc)</literal></para>
+ </entry>
+ <entry>
+ <para>Boolean that determines if <literal>nagle</literal> should be enabled. It should never be set in production systems.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>keepalive_idle</literal></para>
+ <para> <literal>(30,Wc)</literal></para>
+ </entry>
+ <entry>
+ <para>Time (in seconds) that a socket can remain idle before a keepalive probe is sent. Setting this value to zero (0) disables keepalives.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>keepalive_intvl</literal></para>
+ <para> <literal>(2,Wc)</literal></para>
+ </entry>
+ <entry>
+ <para>Time (in seconds) to repeat unanswered keepalive probes. Setting this value to zero (0) disables keepalives.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>keepalive_count</literal></para>
+ <para> <literal>(10,Wc)</literal></para>
+ </entry>
+ <entry>
+ <para>Number of unanswered keepalive probes before pronouncing socket (hence peer) death.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>enable_irq_affinity</literal></para>
+ <para> <literal>(0,Wc)</literal></para>
+ </entry>
+ <entry>
+ <para>Boolean that determines whether to enable IRQ affinity. The default is zero (0).</para>
+ <para>When set, <literal>socklnd</literal> attempts to maximize performance by handling device interrupts and data movement for particular (hardware) interfaces on particular CPUs. This option is not available on all platforms. This option requires an SMP system to exist and produces best performance with multiple NICs. Systems with multiple CPUs and a single NIC may see increase in the performance with this parameter disabled.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>zc_min_frag</literal></para>
+ <para> <literal>(2048,W)</literal></para>
+ </entry>
+ <entry>
+ <para>Determines the minimum message fragment that should be considered for zero-copy sends. Increasing it above the platform's <literal>PAGE_SIZE </literal>disables all zero copy sends. This option is not available on all platforms.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h3">
+ <title>35.2.3 Portals LND (Linux)</title>
+ <para>The Portals LND Linux (<literal>ptllnd</literal>) can be used as a interface layer to communicate with Sandia Portals networking devices. This version is intended to work on Cray XT3 Linux nodes that use Cray Portals as a network transport.</para>
+ <para><emphasis role="bold">Message Buffers</emphasis></para>
+ <para>When <literal>ptllnd</literal> starts up, it allocates and posts sufficient message buffers to allow all expected peers (set by concurrent_peers) to send one unsolicited message. The first message that a peer actually sends is (so-called) "<literal>HELLO</literal>" message, used to negotiate how much additional buffering to setup (typically 8 messages). If 10000 peers actually exist, then enough buffers are posted for 80000 messages.</para>
+ <para>The maximum message size is set by the <literal>max_msg_size</literal> module parameter (default value is 512). This parameter sets the bulk transfer breakpoint. Below this breakpoint, payload data is sent in the message itself. Above this breakpoint, a buffer descriptor is sent and the receiver gets the actual payload.</para>
+ <para>The buffer size is set by the <literal>rxb_npages</literal> module parameter (default value is <literal>1</literal>). The default conservatively avoids allocation problems due to kernel memory fragmentation. However, increasing this value to 2 is probably not risky.</para>
+ <para>The <literal>ptllnd</literal> also keeps an additional <literal>rxb_nspare</literal> buffers (default value is 8) posted to account for full buffers being handled.</para>
+ <para>Assuming a 4K page size with 10000 peers, 1258 buffers can be expected to be posted at startup, increasing to a maximum of 10008 as peers that are actually connected. By doubling <literal>rxb_npages</literal> halving <literal>max_msg_size</literal>, this number can be reduced by a factor of 4.</para>
+ <para><emphasis role="bold">ME/MD Queue Length</emphasis></para>
+ <para>The <literal>ptllnd</literal> uses a single portal set by the portal module parameter (default value of 9) for both message and bulk buffers. Message buffers are always attached with <literal>PTL_INS_AFTER</literal> and match anything sent with "message" matchbits. Bulk buffers are always attached with <literal>PTL_INS_BEFORE</literal> and match only specific matchbits for that particular bulk transfer.</para>
+ <para>This scheme assumes that the majority of ME/MDs posted are for "message" buffers, and that the overhead of searching through the preceding "bulk" buffers is acceptable. Since the number of "bulk" buffers posted at any time is also dependent on the bulk transfer breakpoint set by <literal>max_msg_size</literal>, this seems like an issue worth measuring at scale.</para>
+ <para><emphasis role="bold">TX Descriptors</emphasis></para>
+ <para>The <literal>ptllnd</literal> has a pool of so-called "tx descriptors", which it uses not only for outgoing messages, but also to hold state for bulk transfers requested by incoming messages. This pool should scale with the total number of peers.</para>
+ <para>To enable the building of the Portals LND (<literal>ptllnd.ko</literal>) configure with this option:</para>
+ <screen>./configure --with-portals=<path-to-portals-headers></screen>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Variable</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>ntx</literal></para>
+ <para> <literal>(256)</literal></para>
+ </entry>
+ <entry>
+ <para>Total number of messaging descriptors.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>concurrent_peers</literal></para>
+ <para> <literal>(1152)</literal></para>
+ </entry>
+ <entry>
+ <para>Maximum number of concurrent peers. Peers that attempt to connect beyond the maximum are not allowed.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>peer_hash_table_size</literal></para>
+ <para> <literal>(101)</literal></para>
+ </entry>
+ <entry>
+ <para>Number of hash table slots for the peers. This number should scale with <literal>concurrent_peers</literal>. The size of the peer hash table is set by the module parameter <literal>peer_hash_table_size</literal> which defaults to a value of 101. This number should be prime to ensure the peer hash table is populated evenly. It is advisable to increase this value to 1001 for ~10000 peers.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>cksum</literal></para>
+ <para> <literal>(0)</literal></para>
+ </entry>
+ <entry>
+ <para>Set to non-zero to enable message (not RDMA) checksums for outgoing packets. Incoming packets are always check-summed if necessary, independent of this value.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>timeout</literal></para>
+ <para> <literal>(50)</literal></para>
+ </entry>
+ <entry>
+ <para>Amount of time (in seconds) that a request can linger in a peers-active queue before the peer is considered dead.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>portal</literal></para>
+ <para> <literal>(9)</literal></para>
+ </entry>
+ <entry>
+ <para>Portal ID to use for the <literal>ptllnd</literal> traffic.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>rxb_npages</literal></para>
+ <para> <literal>(64 * #cpus)</literal></para>
+ </entry>
+ <entry>
+ <para>Number of pages in an RX buffer.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>credits</literal></para>
+ <para> <literal>(128)</literal></para>
+ </entry>
+ <entry>
+ <para>Maximum total number of concurrent sends that are outstanding to a single peer at a given time.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>peercredits</literal></para>
+ <para> <literal>(8)</literal></para>
+ </entry>
+ <entry>
+ <para>Maximum number of concurrent sends that are outstanding to a single peer at a given time.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>max_msg_size</literal></para>
+ <para> <literal>(512)</literal></para>
+ </entry>
+ <entry>
+ <para>Maximum immediate message size. This MUST be the same on all nodes in a cluster. A peer that connects with a different <literal>max_msg_size</literal> value will be rejected.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h3">
+ <title>35.2.4 MX LND</title>
+ <para><literal>MXLND</literal> supports a number of load-time parameters using Linux's module parameter system. The following variables are available:</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Variable</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>n_waitd</literal></para>
+ </entry>
+ <entry>
+ <para>Number of completion daemons.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>max_peers</literal></para>
+ </entry>
+ <entry>
+ <para>Maximum number of peers that may connect.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>cksum</literal></para>
+ </entry>
+ <entry>
+ <para>Enables small message (< 4 KB) checksums if set to a non-zero value.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ntx</literal></para>
+ </entry>
+ <entry>
+ <para>Number of total tx message descriptors.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>credits</literal></para>
+ </entry>
+ <entry>
+ <para>Number of concurrent sends to a single peer.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>board</literal></para>
+ </entry>
+ <entry>
+ <para>Index value of the Myrinet board (NIC).</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ep_id</literal></para>
+ </entry>
+ <entry>
+ <para>MX endpoint ID.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>polling</literal></para>
+ </entry>
+ <entry>
+ <para>Use zero (0) to block (wait). A value > 0 will poll that many times before blocking.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>hosts</literal></para>
+ </entry>
+ <entry>
+ <para>IP-to-hostname resolution file.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <para>Of the described variables, only hosts is required. It must be the absolute path to the MXLND hosts file.</para>
+ <para>For example:</para>
+ <screen>options kmxlnd hosts=/etc/hosts.mxlnd</screen>
+ <para>The file format for the hosts file is:</para>
+ <screen>IP HOST BOARD EP_ID</screen>
+ <para>The values must be space and/or tab separated where:</para>
+ <para><literal>IP</literal> is a valid IPv4 address</para>
+ <para><literal>HOST</literal> is the name returned by <literal>`hostname`</literal> on that machine</para>
+ <para><literal>BOARD</literal> is the index of the Myricom NIC (0 for the first card, etc.)</para>
+ <para><literal>EP_ID</literal> is the MX endpoint ID</para>
+ <para>To obtain the optimal performance for your platform, you may want to vary the remaining options.</para>
+ <para><literal>n_waitd(1)</literal> sets the number of threads that process completed MX requests (sends and receives).</para>
+ <para><literal>max_peers(1024)</literal> tells MXLND the upper limit of machines that it will need to communicate with. This affects how many receives it will pre-post and each receive will use one page of memory. Ideally, on clients, this value will be equal to the total number of Lustre servers (MDS and OSS). On servers, it needs to equal the total number of machines in the storage system. cksum (0) turns on small message checksums. It can be used to aid in troubleshooting. MX also provides an optional checksumming feature which can check all messages (large and small). For details, see the MX README.</para>
+ <para><literal>ntx(256)</literal> is the number of total sends in flight from this machine. In actuality, MXLND reserves half of them for connect messages so make this value twice as large as you want for the total number of sends in flight.</para>
+ <para><literal>credits(8)</literal> is the number of in-flight messages for a specific peer. This is part of the flow-control system in Lustre. Increasing this value may improve performance but it requires more memory because each message requires at least one page.</para>
+ <para><literal>board(0)</literal> is the index of the Myricom NIC. Hosts can have multiple Myricom NICs and this identifies which one MXLND should use. This value must match the board value in your MXLND hosts file for this host.</para>
+ <para><literal>ep_id(3)</literal> is the MX endpoint ID. Each process that uses MX is required to have at least one MX endpoint to access the MX library and NIC. The ID is a simple index starting at zero (0). This value must match the endpoint ID value in your MXLND hosts file for this host.</para>
+ <para><literal>polling(0)</literal> determines whether this host will poll or block for MX request completions. A value of 0 blocks and any positive value will poll that many times before blocking. Since polling increases CPU usage, we suggest that you set this to zero (0) on the client and experiment with different values for servers.</para>
+ </section>
+ </section>
</chapter>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='lustreprogramminginterfaces'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lustreprogramminginterfaces">
<info>
- <title xml:id='lustreprogramminginterfaces.title'>Lustre Programming Interfaces</title>
+ <title xml:id="lustreprogramminginterfaces.title">Lustre Programming Interfaces</title>
</info>
<para>This chapter describes public programming interfaces to control various aspects of Lustre from userspace. These interfaces are generally not guaranteed to remain unchanged over time, although we will make an effort to notify the user community well in advance of major changes. This chapter includes the following section:</para>
-
- <itemizedlist><listitem>
+ <itemizedlist>
+ <listitem>
<para><xref linkend="dbdoclet.50438291_32926"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438291_73963"/></para>
</listitem>
-
-</itemizedlist>
-
- <note><para>Lustre programming interface man pages are found in the lustre/doc folder.</para></note>
-
- <section xml:id="dbdoclet.50438291_32926">
- <title>33.1 User/Group <anchor xml:id="dbdoclet.50438291_marker-1293215" xreflabel=""/>Cache Upcall</title>
- <para>This section describes user and group upcall.</para>
- <note><para>For information on a universal UID/GID, see <link xl:href="InstallingLustre.html#50438261_19503">Environmental Requirements</link>.</para></note>
-
- <section remap="h3">
- <title>33.1.1 Name</title>
- <para>Use /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall to look up a given user's group membership.</para>
- </section>
- <section remap="h3">
- <title>33.1.2 Description</title>
- <para>The group upcall file contains the path to an executable that, when installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data data structure (see <link xl:href="LustreProgrammingInterfaces.html#50438291_33759">Data Structures</link>) and write it to the /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info pseudo-file.</para>
- <para>For a sample upcall program, see lustre/utils/l_getgroups.c in the Lustre source distribution.</para>
- <section remap="h4">
- <title>33.1.2.1 Primary and Secondary Groups</title>
- <para>The mechanism for the primary/secondary group is as follows:</para>
- <itemizedlist><listitem>
- <para> The MDS issues an upcall (set per MDS) to map the numeric UID to the supplementary group(s).</para>
- </listitem>
-
-<listitem>
- <para> If there is no upcall or if there is an upcall and it fails, supplementary groups will be added as supplied by the client (as they are now).</para>
- </listitem>
-
-<listitem>
- <para> The default upcall is /usr/sbin/l_getidentity, which can interact with the user/group database to obtain UID/GID/suppgid. The user/group database depends on authentication configuration, and can be local /etc/passwd, NIS, LDAP, etc. If necessary, the administrator can use a parse utility to set /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall. If the upcall interface is set to NONE, then upcall is disabled. The MDS uses the UID/GID/suppgid supplied by the client.</para>
- </listitem>
-
-<listitem>
- <para> The default group upcall is set by mkfs.lustre. Use tunefs.lustre --param or echo{path}>/proc/fs/lustre/mds/{mdsname}/group_upcall</para>
- </listitem>
-
-<listitem>
- <para> The Lustre administrator can specify permissions for a specific UID by configuring /etc/lustre/perm.conf on the MDS. As commented in lustre/utils/l_getidentity.c</para>
- </listitem>
-
-</itemizedlist>
- <screen>/** permission file format is like this: * {nid} {uid} {perms} * * '*' nid \
-means any nid* '*' uid means any uid* the valid values for perms are:* setu\
-id/setgid/setgrp/rmtacl -- enable corresponding perm* nosetuid/nosetgid/nos\
-etgrp/normtacl -- disable corresponding perm* they can be listed together, \
-seperated by ',',* when perm and noperm are in the same line (item), noperm\
- is preferential,* when they are in different lines (items), the latter is \
-preferential,* '*' nid is as default perm, and is not preferential.*/
-</screen>
- <para>Currently, rmtacl/normtacl can be ignored (part of security functionality), and used for remote clients. The /usr/sbin/l_getidentity utility can parse /etc/lustre/perm.conf to obtain permission mask for specified UID.</para>
- <itemizedlist><listitem>
- <para> To avoid repeated upcalls, the MDS caches supplemental group information. Use /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_expire to set the cache time (default is 600 seconds). The kernel waits for the upcall to complete (at most, 5 seconds) and takes the "failure" behavior as described. Set the wait time in /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_acquire_expire (default is 15 seconds). Cached entries are flushed by writing to /proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_flush.</para>
- </listitem>
-
-</itemizedlist>
- </section>
- </section>
- <section remap="h3">
- <title>33.1.3 Parameters</title>
- <itemizedlist><listitem>
- <para> Name of the MDS service</para>
+ </itemizedlist>
+ <note>
+ <para>Lustre programming interface man pages are found in the <literal>lustre/doc</literal> folder.</para>
+ </note>
+ <section xml:id="dbdoclet.50438291_32926">
+ <title>33.1 User/Group Cache Upcall</title>
+ <para>This section describes user and group upcall.</para>
+ <note>
+ <para>For information on a universal UID/GID, see <xref linkend="dbdoclet.50438261_19503"/>.</para>
+ </note>
+ <section remap="h3">
+ <title>33.1.1 Name</title>
+ <para>Use <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall</literal> to look up a given user's group membership.</para>
+ </section>
+ <section remap="h3">
+ <title>33.1.2 Description</title>
+ <para>The group upcall file contains the path to an executable that, when installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the <literal>mds_grp_downcall_data</literal> data structure (see <xref linkend="dbdoclet.50438291_33759"/>) and write it to the <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_info</literal> pseudo-file.</para>
+ <para>For a sample upcall program, see <literal>lustre/utils/l_getgroups.c</literal> in the Lustre source distribution.</para>
+ <section remap="h4">
+ <title>33.1.2.1 Primary and Secondary Groups</title>
+ <para>The mechanism for the primary/secondary group is as follows:</para>
+ <itemizedlist>
+ <listitem>
+ <para>The MDS issues an upcall (set per MDS) to map the numeric UID to the supplementary group(s).</para>
+ </listitem>
+ <listitem>
+ <para>If there is no upcall or if there is an upcall and it fails, supplementary groups will be added as supplied by the client (as they are now).</para>
+ </listitem>
+ <listitem>
+ <para>The default upcall is <literal>/usr/sbin/l_getidentity</literal>, which can interact with the user/group database to obtain UID/GID/suppgid. The user/group database depends on authentication configuration, and can be local <literal>/etc/passwd</literal>, NIS, LDAP, etc. If necessary, the administrator can use a parse utility to set <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_upcall</literal>. If the upcall interface is set to NONE, then upcall is disabled. The MDS uses the UID/GID/suppgid supplied by the client.</para>
</listitem>
-
-<listitem>
- <para> Numeric UID</para>
+ <listitem>
+ <para>The default group upcall is set by mkfs.lustre. Use <literal>tunefs.lustre --param</literal> or <literal>echo{path}>/proc/fs/lustre/mds/{mdsname}/group_upcall</literal></para>
</listitem>
-
-</itemizedlist>
+ <listitem>
+ <para>The Lustre administrator can specify permissions for a specific UID by configuring <literal>/etc/lustre/perm.conf</literal> on the MDS. As commented in <literal>lustre/utils/l_getidentity.c</literal></para>
+ </listitem>
+ </itemizedlist>
+ <screen>
+/*
+* permission file format is like this:
+* {nid} {uid} {perms}
+*
+* '*' nid means any nid
+* '*' uid means any uid
+* the valid values for perms are:
+* setuid/setgid/setgrp/rmtacl -- enable corresponding perm
+* nosetuid/nosetgid/nosetgrp/normtacl -- disable corresponding perm
+* they can be listed together, seperated by ',',
+* when perm and noperm are in the same line (item), noperm is preferential,
+* when they are in different lines (items), the latter is preferential,
+* '*' nid is as default perm, and is not preferential.*/
+</screen>
+ <para>Currently, <literal>rmtacl/normtacl</literal> can be ignored (part of security functionality), and used for remote clients. The /usr/sbin/l_getidentity utility can parse <literal>/etc/lustre/perm.conf</literal> to obtain permission mask for specified UID.</para>
+ <itemizedlist>
+ <listitem>
+ <para> To avoid repeated upcalls, the MDS caches supplemental group information. Use <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_expire</literal> to set the cache time (default is 600 seconds). The kernel waits for the upcall to complete (at most, 5 seconds) and takes the "failure" behavior as described. Set the wait time in <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_acquire_expire</literal> (default is 15 seconds). Cached entries are flushed by writing to <literal>/proc/fs/lustre/mdt/${FSNAME}-MDT{xxxx}/identity_flush</literal>.</para>
+ </listitem>
+ </itemizedlist>
</section>
- <section remap="h3">
- <title>33.1.4 <anchor xml:id="dbdoclet.50438291_33759" xreflabel=""/>Data Structures</title>
- <screen>struct identity_downcall_data {
+ </section>
+ <section remap="h3">
+ <title>33.1.3 Parameters</title>
+ <itemizedlist>
+ <listitem>
+ <para> Name of the MDS service</para>
+ </listitem>
+ <listitem>
+ <para> Numeric UID</para>
+ </listitem>
+ </itemizedlist>
+ </section>
+ <section remap="h3">
+ <title>33.1.4 <anchor xml:id="dbdoclet.50438291_33759" xreflabel=""/>Data Structures</title>
+ <screen>struct identity_downcall_data {
__u32 idd_magic;
__u32 idd_err;
__u32 idd_uid;
struct perm_downcall_data idd_perms[N_PERMS_MAX];
__u32 idd_ngroups;
__u32 idd_groups[0];
-};
-</screen>
- </section>
+};</screen>
+ </section>
+ </section>
+ <section xml:id="dbdoclet.50438291_73963">
+ <title>33.2 <literal>l_getgroups</literal><anchor xml:id="dbdoclet.50438291_marker-1294565" xreflabel=""/> Utility</title>
+ <para>The <literal>l_getgroups</literal> utility handles Lustre user/group cache upcall.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>l_getgroups [-v] [-d|mdsname] uid]
+l_getgroups [-v] -s</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The group upcall file contains the path to an executable that, when properly installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the <literal>mds_grp_downcall_data</literal> data structure (see Data structures) and write it to the <literal>/proc/fs/lustre/mds/mds-service/group_info</literal> pseudo-file.</para>
+ <para>l_getgroups is the reference implementation of the user/group cache upcall.</para>
+ </section>
+ <section remap="h5">
+ <title>Files</title>
+ <para><literal>/proc/fs/lustre/mds/mds-service/group_upcall</literal></para>
</section>
- <section xml:id="dbdoclet.50438291_73963">
- <title>33.2 l_getgroups<anchor xml:id="dbdoclet.50438291_marker-1294565" xreflabel=""/> Utility</title>
- <para>The l_getgroups utility handles Lustre user/group cache upcall.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>l_getgroups [-v] [-d|mdsname] uid]
-l_getgroups [-v] -s
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The group upcall file contains the path to an executable that, when properly installed, is invoked to resolve a numeric UID to a group membership list. This utility should complete the mds_grp_downcall_data data structure (see Data structures) and write it to the /proc/fs/lustre/mds/mds-service/group_info pseudo-file.</para>
- <para>l_getgroups is the reference implementation of the user/group cache upcall.</para>
- </section>
- <section remap="h5">
- <title>Files</title>
- <para>/proc/fs/lustre/mds/mds-service/group_upcall</para>
- </section>
</section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<preface xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US">
+<!-- This document was created with Syntext Serna Free. --><preface xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US">
<title>Preface</title>
<para>This operations manual provides detailed information and procedures to install, configure and tune the Lustre file system. The manual covers topics such as failover, quotas, striping and bonding. The Lustre manual also contains troubleshooting information and tips to improve Lustre operation and performance.</para>
<section remap="h2">
- <title>
-
- </title>
+ <title>About this Document</title>
+ <para>This document is maintained by Whamcloud, Inc in Docbook format. The canonical version is available at <link xml:href="http://wiki.whamcloud.com">http://wiki.whamcloud.com/</link>. </para>
<section remap="h2">
<title>UNIX Commands</title>
<para>This document might not contain information about basic UNIX commands and procedures such as shutting down the system, booting the system, and configuring devices. Refer to the following for this information:</para>
</informaltable>
</section>
<section remap="h2">
- <title><anchor xml:id="dbdoclet.50438247_43930" xreflabel=""/>Related Documentation</title>
+ <title>Related Documentation</title>
<para>The documents listed as online are available at:</para>
<para><link xl:href="http://wiki.whamcloud.com/display/PUB/Documentation">http://wiki.whamcloud.com/display/PUB/Documentation</link></para>
<informaltable frame="all">
<para> Support <link xl:href="http://www.whamcloud.com/">http://www.whamcloud.com/</link></para>
</listitem>
<listitem>
- <para> Training <link xl:href="http://www.whamcloud.com/">http://www.whamcloud.com/</link><anchor xml:id="dbdoclet.50438247_88598" xreflabel=""/><anchor xml:id="dbdoclet.50438247_83191" xreflabel=""/></para>
+ <para> Training <link xl:href="http://www.whamcloud.com/">http://www.whamcloud.com/</link></para>
</listitem>
</itemizedlist>
- <para> </para>
</section>
</section>
</preface>
-<?xml version="1.0" encoding="UTF-8"?>
-<chapter version="5.0" xml:lang="en-US" xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" xml:id='settinglustreproperties'>
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="settinglustreproperties">
<info>
- <title xml:id='settinglustreproperties.title'>Setting Lustre Properties in a C Program (llapi)</title>
+ <title xml:id="settinglustreproperties.title">Setting Lustre Properties in a C Program (<literal>llapi</literal>)</title>
</info>
- <para>This chapter describes the llapi library of commands used for setting Lustre file properties within a C program running in a cluster environment, such as a data processing or MPI application. The commands described in this chapter are:</para>
- <itemizedlist><listitem>
+ <para>This chapter describes the <literal>llapi</literal> library of commands used for setting Lustre file properties within a C program running in a cluster environment, such as a data processing or MPI application. The commands described in this chapter are:</para>
+ <itemizedlist>
+ <listitem>
<para><xref linkend="dbdoclet.50438215_30970"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438215_50149"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438215_86607"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438215_12433"/></para>
</listitem>
-
-<listitem>
+ <listitem>
<para><xref linkend="dbdoclet.50438215_15718"/></para>
</listitem>
+ </itemizedlist>
+ <note>
+ <para>Lustre programming interface man pages are found in the <literal>lustre/doc</literal> folder.</para>
+ </note>
+ <section xml:id="dbdoclet.50438215_30970">
+ <title>34.1 <literal>llapi_file_create</literal></title>
+ <para>Use <literal>llapi_file_create</literal> to set Lustre properties for a new file.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>#include <lustre/liblustreapi.h>
+#include <lustre/lustre_user.h>
-</itemizedlist>
-
- <note><para>Lustre programming interface man pages are found in the lustre/doc folder.</para></note>
-
- <section xml:id="dbdoclet.50438215_30970">
- <title>34.1 llapi_file_create</title>
- <para>Use llapi_file_create to set Lustre properties for a new file.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>#include <lustre/liblustreapi.h>#include <lustre/lustre_user.h>
-int llapi_file_create(char *name, long stripe_size, int stripe_offset, int \
-stripe_count, int stripe_pattern);
+int llapi_file_create(char *name, long stripe_size, int stripe_offset, int stripe_count, int stripe_pattern);
</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The llapi_file_create() function sets a file descriptor's Lustre striping information. The file descriptor is then accessed with open ().</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">llapi_file_create()</emphasis></para></entry>
- <entry><para> If the file already exists, this parameter returns to 'EEXIST'. If the stripe parameters are invalid, this parameter returns to 'EINVAL'.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stripe_size</emphasis></para></entry>
- <entry><para> This value must be an even multiple of system page size, as shown by getpagesize (). The default Lustre stripe size is 4MB.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stripe_offset</emphasis></para></entry>
- <entry><para> Indicates the starting OST for this file.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stripe_count</emphasis></para></entry>
- <entry><para> Indicates the number of OSTs that this file will be striped across.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stripe_pattern</emphasis></para></entry>
- <entry><para> Indicates the RAID pattern.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- <note><para>Currently, only RAID 0 is supported. To use the system defaults, set these values: stripe_size = 0, stripe_offset = -1, stripe_count = 0, stripe_pattern = 0</para></note>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <para>System default size is 4 MB.</para>
- <screen>char *tfile = TESTFILE;
-int stripe_size = 65536
-</screen>
- <para>To start at default, run:</para>
- <screen>int stripe_offset = -1
-</screen>
- <para>To start at the default, run:</para>
- <screen>int stripe_count = 1
-</screen>
- <para>To set a single stripe for this example, run:</para>
- <screen>int stripe_pattern = 0
-</screen>
- <para>Currently, only RAID 0 is supported.</para>
- <screen>int stripe_pattern = 0;
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The <literal>llapi_file_create()</literal> function sets a file descriptor's Lustre striping information. The file descriptor is then accessed with <literal>open()</literal>.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>llapi_file_create()</literal></para>
+ </entry>
+ <entry>
+ <para>If the file already exists, this parameter returns to '<literal>EEXIST</literal>'. If the stripe parameters are invalid, this parameter returns to '<literal>EINVAL</literal>'.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_size</literal></para>
+ </entry>
+ <entry>
+ <para>This value must be an even multiple of system page size, as shown by <literal>getpagesize()</literal>. The default Lustre stripe size is 4MB.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_offset</literal></para>
+ </entry>
+ <entry>
+ <para>Indicates the starting OST for this file.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_count</literal></para>
+ </entry>
+ <entry>
+ <para>Indicates the number of OSTs that this file will be striped across.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_pattern</literal></para>
+ </entry>
+ <entry>
+ <para>Indicates the RAID pattern.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ <note>
+ <para>Currently, only RAID 0 is supported. To use the system defaults, set these values: <literal>stripe_size</literal> = 0, <literal>stripe_offset</literal> = -1, <literal>stripe_count</literal> = 0, <literal>stripe_pattern</literal> = 0</para>
+ </note>
+ </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <para>System default size is 4 MB.</para>
+ <screen>char *tfile = TESTFILE;
+int stripe_size = 65536</screen>
+ <para>To start at default, run:</para>
+ <screen>int stripe_offset = -1</screen>
+ <para>To start at the default, run:</para>
+ <screen>int stripe_count = 1</screen>
+ <para>To set a single stripe for this example, run:</para>
+ <screen>int stripe_pattern = 0</screen>
+ <para>Currently, only RAID 0 is supported.</para>
+ <screen>int stripe_pattern = 0;
int rc, fd;
-rc = llapi_file_create(tfile, stripe_size,stripe_offset, stripe_count,strip\
-e_pattern);
-</screen>
- <para>Result code is inverted, you may return with 'EINVAL' or an ioctl error.</para>
- <screen>if (rc) {
-fprintf(stderr,"llapi_file_create failed: %d (%s) 0, rc, strerror(-rc));retu\
-rn -1; }
-</screen>
- <para>llapi_file_create closes the file descriptor. You must re-open the descriptor. To do this, run:</para>
- <screen>fd = open(tfile, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE, 0644); if (fd < 0) \\
- { fprintf(stderr, "Can't open %s file: %s0, tfile,
+rc = llapi_file_create(tfile, stripe_size,stripe_offset, stripe_count,stripe_pattern);</screen>
+ <para>Result code is inverted, you may return with '<literal>EINVAL</literal>' or an ioctl error.</para>
+ <screen>if (rc) {
+fprintf(stderr,"llapi_file_create failed: %d (%s) 0, rc, strerror(-rc));return -1; }</screen>
+ <para><literal>llapi_file_create</literal> closes the file descriptor. You must re-open the descriptor. To do this, run:</para>
+ <screen>fd = open(tfile, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE, 0644); if (fd < 0) \ {
+fprintf(stderr, "Can't open %s file: %s0, tfile,
str-
error(errno));
return -1;
-}
-</screen>
- </section>
+}</screen>
</section>
- <section xml:id="dbdoclet.50438215_50149">
- <title>34.2 llapi_file_get_stripe</title>
- <para>Use llapi_file_get_stripe to get striping information for a file or directory on a Lustre file system.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>#include <sys/types.h>
+ </section>
+ <section xml:id="dbdoclet.50438215_50149">
+ <title>34.2 llapi_file_get_stripe</title>
+ <para>Use <literal>llapi_file_get_stripe</literal> to get striping information for a file or directory on a Lustre file system.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <liblustre.h>
#include <lustre/liblustreapi.h>
#include <lustre/lustre_user.h>
-int llapi_file_get_stripe(const char *<emphasis>path</emphasis>, void *<emphasis>lum</emphasis>);
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The llapi_file_get_stripe() function returns striping information for a file or directory <emphasis>path</emphasis> in <emphasis>lum</emphasis> (which should point to a large enough memory region) in one of the following formats:</para>
- <screen>struct lov_user_md_v1 {
+int llapi_file_get_stripe(const char *<emphasis>path</emphasis>, void *<emphasis>lum</emphasis>);</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The <literal>llapi_file_get_stripe()</literal> function returns striping information for a file or directory <emphasis>path</emphasis> in <emphasis>lum</emphasis> (which should point to a large enough memory region) in one of the following formats:</para>
+ <screen>struct lov_user_md_v1 {
__u32 lmm_magic;
__u32 lmm_pattern;
__u64 lmm_object_id;
__u16 lmm_stripe_offset;
char lmm_pool_name[LOV_MAXPOOLNAME];
struct lov_user_ost_data_v1 lmm_objects[0];
-} __attribute__((packed));
-</screen>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">lmm_magic</emphasis></para></entry>
- <entry><para> Specifies the format of the returned striping information. <emphasis role="bold">LOV_MAGIC_V1</emphasis> isused for lov_user_md_v1. <emphasis role="bold">LOV_MAGIC_V3</emphasis> is used for lov_user_md_v3.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">lmm_pattern</emphasis></para></entry>
- <entry><para> Holds the striping pattern. Only <emphasis role="bold">LOV_PATTERN_RAID0</emphasis> is possible in this Lustre version.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">lmm_object_id</emphasis></para></entry>
- <entry><para> Holds the MDS object ID.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">lmm_object_gr</emphasis></para></entry>
- <entry><para> Holds the MDS object group.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">lmm_stripe_size</emphasis></para></entry>
- <entry><para> Holds the stripe size in bytes.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">lmm_stripe_count</emphasis></para></entry>
- <entry><para> Holds the number of OSTs over which the file is striped.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">lmm_stripe_offset</emphasis></para></entry>
- <entry><para> Holds the OST index from which the file starts.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">lmm_pool_name</emphasis></para></entry>
- <entry><para> Holds the OST pool name to which the file belongs.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">lmm_objects</emphasis></para></entry>
- <entry><para> An array of lmm_stripe_count members containing per OST file information in</para><para>the following format:</para><para>struct lov_user_ost_data_v1 {</para><para>__u64 l_object_id;</para><para>__u64 l_object_seq;</para><para>__u32 l_ost_gen;</para><para>__u32 l_ost_idx;</para><para>} __attribute__((packed));</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">l_object_id</emphasis></para></entry>
- <entry><para> Holds the OST's object ID.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">l_object_seq</emphasis></para></entry>
- <entry><para> Holds the OST's object group.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">l_ost_gen</emphasis></para></entry>
- <entry><para> Holds the OST's index generation.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">l_ost_idx</emphasis></para></entry>
- <entry><para> Holds the OST's index in LOV.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Return Values</title>
- <para>llapi_file_get_stripe() returns:</para>
- <para><emphasis role="bold">0</emphasis> On success</para>
- <para><emphasis role="bold">!= 0</emphasis> On failure, <emphasis>errno</emphasis> is set appropriately</para>
- </section>
- <section remap="h5">
- <title>Errors</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Errors</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">ENOMEM</emphasis></para></entry>
- <entry><para> <emphasis role="bold">Failed to allocate memory</emphasis></para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ENAMETOOLONG</emphasis></para></entry>
- <entry><para> Path was too long</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ENOENT</emphasis></para></entry>
- <entry><para> Path does not point to a file or directory</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ENOTTY</emphasis></para></entry>
- <entry><para> Path does not point to a Lustre file system</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">EFAULT</emphasis></para></entry>
- <entry><para> Memory region pointed by lum is not properly mapped</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Examples</title>
- <screen>#include <sys/vfs.h>
+} __attribute__((packed));</screen>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>lmm_magic</literal></para>
+ </entry>
+ <entry>
+ <para>Specifies the format of the returned striping information. <literal>LOV_MAGIC_V1</literal> isused for lov_user_md_v1. LOV_MAGIC_V3 is used for <literal>lov_user_md_v3</literal>.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>lmm_pattern</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the striping pattern. Only <literal>LOV_PATTERN_RAID0</literal> is possible in this Lustre version.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>lmm_object_id</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the MDS object ID.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>lmm_object_gr</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the MDS object group.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>lmm_stripe_size</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the stripe size in bytes.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>lmm_stripe_count</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the number of OSTs over which the file is striped.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>lmm_stripe_offset</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the OST index from which the file starts.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>lmm_pool_name</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the OST pool name to which the file belongs.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>lmm_objects</literal></para>
+ </entry>
+ <entry>
+ <para>An array of <literal>lmm_stripe_count</literal> members containing per OST file information in</para>
+ <para>the following format:</para>
+ <screen>struct lov_user_ost_data_v1 {
+ __u64 l_object_id;
+ __u64 l_object_seq;
+ __u32 l_ost_gen;
+ __u32 l_ost_idx;
+ } __attribute__((packed));</screen>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>l_object_id</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the OST's object ID.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>l_object_seq</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the OST's object group.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>l_ost_gen</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the OST's index generation.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>l_ost_idx</literal></para>
+ </entry>
+ <entry>
+ <para>Holds the OST's index in LOV.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Return Values</title>
+ <para><literal>llapi_file_get_stripe()</literal> returns:</para>
+ <para><literal>0</literal> On success</para>
+ <para><literal>!= 0</literal> On failure, <literal>errno</literal> is set appropriately</para>
+ </section>
+ <section remap="h5">
+ <title>Errors</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Errors</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>ENOMEM</literal></para>
+ </entry>
+ <entry>
+ <para>Failed to allocate memory</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ENAMETOOLONG</literal></para>
+ </entry>
+ <entry>
+ <para>Path was too long</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ENOENT</literal></para>
+ </entry>
+ <entry>
+ <para>Path does not point to a file or directory</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ENOTTY</literal></para>
+ </entry>
+ <entry>
+ <para>Path does not point to a Lustre file system</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>EFAULT</literal></para>
+ </entry>
+ <entry>
+ <para>Memory region pointed by lum is not properly mapped</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Examples</title>
+ <screen>#include <sys/vfs.h>
#include <liblustre.h>
#include <lnet/lnetctl.h>
#include <obd.h>
if (lum_file != NULL)
free(lum_file);
return rc;
-}
-</screen>
- </section>
+}</screen>
</section>
- <section xml:id="dbdoclet.50438215_86607">
- <title>34.3 llapi_file_open</title>
- <para>The llapi_file_open command opens (or creates) a file or device on a Lustre filesystem.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>#include <sys/types.h>
+ </section>
+ <section xml:id="dbdoclet.50438215_86607">
+ <title>34.3 <literal>llapi_file_open</literal></title>
+ <para>The <literal>llapi_file_open</literal> command opens (or creates) a file or device on a Lustre filesystem.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <liblustre.h>
int <emphasis>stripe_offset</emphasis>, int <emphasis>stripe_count</emphasis>,
int <emphasis>stripe_pattern</emphasis>);
</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The llapi_file_create() call is equivalent to the llapi_file_open call with <emphasis>flags</emphasis> equal to O_CREAT|O_WRONLY and <emphasis>mode</emphasis> equal to 0644, followed by file close.</para>
- <para>llapi_file_open() opens a file with a given name on a Lustre filesystem.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">flags</emphasis></para></entry>
- <entry><para> Can be a combination of O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_EXCL, O_NOCTTY, O_TRUNC, O_APPEND, O_NONBLOCK, O_SYNC, FASYNC, O_DIRECT, O_LARGEFILE, O_DIRECTORY, O_NOFOLLOW, O_NOATIME.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">mode</emphasis></para></entry>
- <entry><para> Specifies the permission bits to be used for a new file when O_CREAT is used.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stripe_size</emphasis></para></entry>
- <entry><para> Specifies stripe size (in bytes). Should be multiple of 64 KB, not exceeding 4 GB.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stripe_offset</emphasis></para></entry>
- <entry><para> Specifies an OST index from which the file should start. The default value is -1.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stripe_count</emphasis></para></entry>
- <entry><para> Specifies the number of OSTs to stripe the file across. The default value is -1.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">stripe_pattern</emphasis></para></entry>
- <entry><para> Specifies the striping pattern. In this version of Lustre, only LOV_PATTERN_RAID0 is available. The default value is 0.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Return Values</title>
- <para>llapi_file_open() and llapi_file_create() return:</para>
- <para><emphasis role="bold">>=0</emphasis> On success, for llapi_file_open the return value is a file descriptor</para>
- <para><emphasis role="bold"><0</emphasis> On failure, the absolute value is an error code</para>
- </section>
- <section remap="h5">
- <title>Errors</title>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Errors</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">EINVAL</emphasis></para></entry>
- <entry><para> <emphasis role="bold">stripe_size</emphasis> or <emphasis role="bold">stripe_offset</emphasis> or <emphasis role="bold">stripe_count</emphasis> or <emphasis role="bold">stripe_pattern</emphasis> is invalid.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">EEXIST</emphasis></para></entry>
- <entry><para> Striping information has already been set and cannot be altered; <emphasis role="bold">name</emphasis> already exists.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">EALREADY</emphasis></para></entry>
- <entry><para> Striping information has already been set and cannot be altered</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ENOTTY</emphasis></para></entry>
- <entry><para> <emphasis role="bold">name</emphasis> may not point to a Lustre filesystem.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Example</title>
- <screen>#include <sys/types.h>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The <literal>llapi_file_create()</literal> call is equivalent to the <literal>llapi_file_open</literal> call with <emphasis>flags</emphasis> equal to <literal>O_CREAT|O_WRONLY</literal> and <emphasis>mode</emphasis> equal to <literal>0644</literal>, followed by file close.</para>
+ <para><literal>llapi_file_open()</literal> opens a file with a given name on a Lustre filesystem.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>flags</literal></para>
+ </entry>
+ <entry>
+ <para>Can be a combination of <literal>O_RDONLY</literal>, <literal>O_WRONLY</literal>, <literal>O_RDWR</literal>, <literal>O_CREAT</literal>, <literal>O_EXCL</literal>, <literal>O_NOCTTY</literal>, <literal>O_TRUNC</literal>, <literal>O_APPEND</literal>, <literal>O_NONBLOCK</literal>, <literal>O_SYNC</literal>, <literal>FASYNC</literal>, <literal>O_DIRECT</literal>, <literal>O_LARGEFILE</literal>, <literal>O_DIRECTORY</literal>, <literal>O_NOFOLLOW</literal>, <literal>O_NOATIME</literal>.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>mode</literal></para>
+ </entry>
+ <entry>
+ <para>Specifies the permission bits to be used for a new file when <literal>O_CREAT</literal> is used.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_size</literal></para>
+ </entry>
+ <entry>
+ <para>Specifies stripe size (in bytes). Should be multiple of 64 KB, not exceeding 4 GB.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_offset</literal></para>
+ </entry>
+ <entry>
+ <para>Specifies an OST index from which the file should start. The default value is -1.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_count</literal></para>
+ </entry>
+ <entry>
+ <para>Specifies the number of OSTs to stripe the file across. The default value is -1.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>stripe_pattern</literal></para>
+ </entry>
+ <entry>
+ <para>Specifies the striping pattern. In this version of Lustre, only <literal>LOV_PATTERN_RAID0</literal> is available. The default value is 0.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Return Values</title>
+ <para><literal>llapi_file_open()</literal> and <literal>llapi_file_create()</literal> return:</para>
+ <para><literal>>=0</literal> On success, for <literal>llapi_file_open</literal> the return value is a file descriptor</para>
+ <para><literal><0</literal> On failure, the absolute value is an error code</para>
+ </section>
+ <section remap="h5">
+ <title>Errors</title>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Errors</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>EINVAL</literal></para>
+ </entry>
+ <entry>
+ <para><literal>stripe_size</literal> or <literal>stripe_offset</literal> or <literal>stripe_count</literal> or <literal>stripe_pattern</literal> is invalid.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>EEXIST</literal></para>
+ </entry>
+ <entry>
+ <para> triping information has already been set and cannot be altered; <literal>name</literal> already exists.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>EALREADY</literal></para>
+ </entry>
+ <entry>
+ <para>Striping information has already been set and cannot be altered</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ENOTTY</literal></para>
+ </entry>
+ <entry>
+ <para> <literal>name</literal> may not point to a Lustre filesystem.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5">
+ <title>Example</title>
+ <screen>#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
return -1;
rc = llapi_file_create(argv[1], 1048576, 0, 2, LOV_PATTERN_RAID0);
if (rc < 0) {
- fprintf(stderr, "file creation has failed, %s\n", strerror\
-(-rc));
+ fprintf(stderr, "file creation has failed, %s\n", strerror(-rc));
return -1;
}
printf("%s with stripe size 1048576, striped across 2 OSTs,"
" has been created!\n", argv[1]);
return 0;
-}
-</screen>
- </section>
+}</screen>
</section>
- <section xml:id="dbdoclet.50438215_12433">
- <title>34.4 llapi_quotactl</title>
- <para>Use llapi_quotactl to manipulate disk quotas on a Lustre file system.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>#include <liblustre.h>
+ </section>
+ <section xml:id="dbdoclet.50438215_12433">
+ <title>34.4 <literal>llapi_quotactl</literal></title>
+ <para>Use <literal>llapi_quotact</literal>l to manipulate disk quotas on a Lustre file system.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>#include <liblustre.h>
#include <lustre/lustre_idl.h>
#include <lustre/liblustreapi.h>
#include <lustre/lustre_user.h>
};
struct obd_uuid {
char uuid[40];
-};
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The llapi_quotactl() command manipulates disk quotas on a Lustre file system mount. qc_cmd indicates a command to be applied to UID qc_id or GID qc_id.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Option</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">LUSTRE_Q_QUOTAON</emphasis></para></entry>
- <entry><para> Turns on quotas for a Lustre file system. <emphasis>qc_type</emphasis> is USRQUOTA, GRPQUOTA or UGQUOTA (both user and group quota). The quota files must exist. They are normally created with the llapi_quotacheck call. This call is restricted to the super user privilege.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">LUSTRE_Q_QUOTAOFF</emphasis></para></entry>
- <entry><para> Turns off quotas for a Lustre file system. <emphasis>qc_type</emphasis> is USRQUOTA, GRPQUOTA or UGQUOTA (both user and group quota). This call is restricted to the super user privilege.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">LUSTRE_Q_GETQUOTA</emphasis></para></entry>
- <entry><para> Gets disk quota limits and current usage for user or group <emphasis>qc_id</emphasis>. <emphasis>qc_type</emphasis> is USRQUOTA or GRPQUOTA. <emphasis>uuid</emphasis> may be filled with OBD UUID string to query quota information from a specific node. <emphasis>dqb_valid</emphasis> may be set nonzero to query information only from MDS. If <emphasis>uuid</emphasis> is an empty string and <emphasis>dqb_valid</emphasis> is zero then cluster-wide limits and usage are returned. On return, <emphasis>obd_dqblk</emphasis> contains the requested information (block limits unit is kilobyte). Quotas must be turned on before using this command.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">LUSTRE_Q_SETQUOTA</emphasis></para></entry>
- <entry><para> Sets disk quota limits for user or group <emphasis>qc_id</emphasis>. <emphasis>qc_type</emphasis> is USRQUOTA or GRPQUOTA. <emphasis>dqb_valid</emphasis> must be set to QIF_ILIMITS, QIF_BLIMITS or QIF_LIMITS (both inode limits and block limits) dependent on updating limits. <emphasis>obd_dqblk</emphasis> must be filled with limits values (as set in <emphasis>dqb_valid</emphasis>, block limits unit is kilobyte). Quotas must be turned on before using this command.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">LUSTRE_Q_GETINFO</emphasis></para></entry>
- <entry><para> Gets information about quotas. <emphasis>qc_type</emphasis> is either USRQUOTA or GRPQUOTA. On return, <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">LUSTRE_Q_SETINFO</emphasis></para></entry>
- <entry><para> Sets quota information (like grace times). <emphasis>qc_type</emphasis> is either USRQUOTA or GRPQUOTA. <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version and must be zeroed.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
- <section remap="h5">
- <title>Return Values</title>
- <para>llapi_quotactl() returns:</para>
- <para><emphasis role="bold">0</emphasis> On success</para>
- <para><emphasis role="bold">-1</emphasis> On failure and sets error number (errno) to indicate the error</para>
- </section>
- <section remap="h5">
- <title>Errors</title>
- <para>llapi_quotactl errors are described below.</para>
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname="c1" colwidth="50*"/>
- <colspec colname="c2" colwidth="50*"/>
- <thead>
- <row>
- <entry><para><emphasis role="bold">Errors</emphasis></para></entry>
- <entry><para><emphasis role="bold">Description</emphasis></para></entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry><para> <emphasis role="bold">EFAULT</emphasis></para></entry>
- <entry><para> <emphasis>qctl</emphasis> is invalid.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ENOSYS</emphasis></para></entry>
- <entry><para> Kernel or Lustre modules have not been compiled with the QUOTA option.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ENOMEM</emphasis></para></entry>
- <entry><para> Insufficient memory to complete operation.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ENOTTY</emphasis></para></entry>
- <entry><para> <emphasis>qc_cmd</emphasis> is invalid.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">EBUSY</emphasis></para></entry>
- <entry><para> Cannot process during quotacheck.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ENOENT</emphasis></para></entry>
- <entry><para> <emphasis>uuid</emphasis> does not correspond to OBD or <emphasis>mnt</emphasis> does not exist.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">EPERM</emphasis></para></entry>
- <entry><para> The call is privileged and the caller is not the super user.</para></entry>
- </row>
- <row>
- <entry><para> <emphasis role="bold">ESRCH</emphasis></para></entry>
- <entry><para> No disk quota is found for the indicated user. Quotas have not been turned on for this file system.</para></entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </section>
+};</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The <literal>llapi_quotactl()</literal> command manipulates disk quotas on a Lustre file system mount. qc_cmd indicates a command to be applied to UID <literal>qc_id</literal> or GID <literal>qc_id</literal>.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Option</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>LUSTRE_Q_QUOTAON</literal></para>
+ </entry>
+ <entry>
+ <para>Turns on quotas for a Lustre file system. <emphasis>qc_type</emphasis> is <literal>USRQUOTA</literal>, <literal>GRPQUOTA</literal> or <literal>UGQUOTA</literal> (both user and group quota). The quota files must exist. They are normally created with the <literal>llapi_quotacheck</literal> call. This call is restricted to the super user privilege.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>LUSTRE_Q_QUOTAOFF</literal></para>
+ </entry>
+ <entry>
+ <para>Turns off quotas for a Lustre file system. <emphasis>qc_type</emphasis> is <literal>USRQUOTA</literal>, <literal>GRPQUOTA</literal> or <literal>UGQUOTA</literal> (both user and group quota). This call is restricted to the super user privilege.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>LUSTRE_Q_GETQUOTA</literal></para>
+ </entry>
+ <entry>
+ <para>Gets disk quota limits and current usage for user or group <emphasis>qc_id</emphasis>. <emphasis>qc_type</emphasis> is <literal>USRQUOTA</literal> or <literal>GRPQUOTA</literal>. <emphasis>uuid</emphasis> may be filled with <literal>OBD UUID</literal> string to query quota information from a specific node. <emphasis>dqb_valid</emphasis> may be set nonzero to query information only from MDS. If <emphasis>uuid</emphasis> is an empty string and <emphasis>dqb_valid</emphasis> is zero then cluster-wide limits and usage are returned. On return, <emphasis>obd_dqblk</emphasis> contains the requested information (block limits unit is kilobyte). Quotas must be turned on before using this command.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>LUSTRE_Q_SETQUOTA</literal></para>
+ </entry>
+ <entry>
+ <para>Sets disk quota limits for user or group <emphasis>qc_id</emphasis>. <emphasis>qc_type</emphasis> is <literal>USRQUOTA</literal> or <literal>GRPQUOTA</literal>. <emphasis>dqb_valid</emphasis> must be set to <literal>QIF_ILIMITS</literal>, <literal>QIF_BLIMITS</literal> or <literal>QIF_LIMITS</literal> (both inode limits and block limits) dependent on updating limits. <emphasis>obd_dqblk</emphasis> must be filled with limits values (as set in <emphasis>dqb_valid</emphasis>, block limits unit is kilobyte). Quotas must be turned on before using this command.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>LUSTRE_Q_GETINFO</literal></para>
+ </entry>
+ <entry>
+ <para>Gets information about quotas. <emphasis>qc_type</emphasis> is either <literal>USRQUOTA</literal> or <literal>GRPQUOTA</literal>. On return, <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>LUSTRE_Q_SETINFO</literal></para>
+ </entry>
+ <entry>
+ <para>Sets quota information (like grace times). <emphasis>qc_type</emphasis> is either <literal>USRQUOTA</literal> or <literal>GRPQUOTA</literal>. <emphasis>dqi_igrace</emphasis> is inode grace time (in seconds), <emphasis>dqi_bgrace</emphasis> is block grace time (in seconds), <emphasis>dqi_flags</emphasis> is not used by the current Lustre version and must be zeroed.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
+ </section>
+ <section remap="h5"><title>Return Values</title><para><literal>llapi_quotactl()</literal> returns:</para><para><literal>0</literal> On success</para><para><literal>
+ -1
+</literal> On failure and sets error number (<literal>errno</literal>) to indicate the error</para></section>
+ <section remap="h5">
+ <title>Errors</title>
+ <para><literal>llapi_quotactl</literal> errors are described below.</para>
+ <informaltable frame="all">
+ <tgroup cols="2">
+ <colspec colname="c1" colwidth="50*"/>
+ <colspec colname="c2" colwidth="50*"/>
+ <thead>
+ <row>
+ <entry>
+ <para><emphasis role="bold">Errors</emphasis></para>
+ </entry>
+ <entry>
+ <para><emphasis role="bold">Description</emphasis></para>
+ </entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry>
+ <para> <literal>EFAULT</literal></para>
+ </entry>
+ <entry>
+ <para><emphasis>qctl</emphasis> is invalid.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ENOSYS</literal></para>
+ </entry>
+ <entry>
+ <para>Kernel or Lustre modules have not been compiled with the <literal>QUOTA</literal> option.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ENOMEM</literal></para>
+ </entry>
+ <entry>
+ <para>Insufficient memory to complete operation.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ENOTTY</literal></para>
+ </entry>
+ <entry>
+ <para> <emphasis>qc_cmd</emphasis> is invalid.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>EBUSY</literal></para>
+ </entry>
+ <entry>
+ <para>Cannot process during quotacheck.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ENOENT</literal></para>
+ </entry>
+ <entry>
+ <para> <emphasis>uuid</emphasis> does not correspond to OBD or <emphasis>mnt</emphasis> does not exist.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>EPERM</literal></para>
+ </entry>
+ <entry>
+ <para>The call is privileged and the caller is not the super user.</para>
+ </entry>
+ </row>
+ <row>
+ <entry>
+ <para> <literal>ESRCH</literal></para>
+ </entry>
+ <entry>
+ <para>No disk quota is found for the indicated user. Quotas have not been turned on for this file system.</para>
+ </entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </informaltable>
</section>
- <section xml:id="dbdoclet.50438215_15718">
- <title>34.5 llapi_path2fid</title>
- <para>Use llapi_path2fid to get the FID from the pathname.</para>
- <section remap="h5">
- <title>Synopsis</title>
- <screen>#include <lustre/liblustreapi.h>
+ </section>
+ <section xml:id="dbdoclet.50438215_15718">
+ <title>34.5 <literal>llapi_path2fid</literal></title>
+ <para>Use <literal>llapi_path2fid</literal> to get the FID from the pathname.</para>
+ <section remap="h5">
+ <title>Synopsis</title>
+ <screen>#include <lustre/liblustreapi.h>
#include <lustre/lustre_user.h>
-int llapi_path2fid(const char *path, unsigned long long *seq, unsigned long\
- *oid, unsigned long *ver)
-</screen>
- </section>
- <section remap="h5">
- <title>Description</title>
- <para>The llapi_path2fid function returns the FID (sequence : object ID : version) for the pathname.</para>
- </section>
- <section remap="h5">
- <title>Return Values</title>
- <para>llapi_path2fid returns:</para>
- <para><emphasis role="bold">0</emphasis> On success</para>
- <para>non-zero value On failure</para>
- </section>
+int llapi_path2fid(const char *path, unsigned long long *seq, unsigned long *oid, unsigned long *ver)</screen>
+ </section>
+ <section remap="h5">
+ <title>Description</title>
+ <para>The <literal>llapi_path2fid</literal> function returns the FID (sequence : object ID : version) for the pathname.</para>
+ </section>
+ <section remap="h5">
+ <title>Return Values</title>
+ <para><literal>llapi_path2fid</literal> returns:</para>
+ <para><literal>0</literal> On success</para>
+ <para>non-zero value On failure</para>
</section>
- <section xml:id="dbdoclet.50438215_marker-1297700">
- <title>34.6 Example Using the llapi Library</title>
- <para>Use llapi_file_create to set Lustre properties for a new file. For a synopsis and description of llapi_file_create and examples of how to use it, see <xref linkend="configurationfilesmoduleparameters"/>.</para>
- <para>You can set striping from inside programs like ioctl. To compile the sample program, you need to download libtest.c and liblustreapi.c files from the Lustre source tree.</para>
- <para><emphasis role="bold">A simple C program to demonstrate striping API - libtest.c</emphasis></para>
- <screen>/* -*- mode: c; c-basic-offset: 8; indent-tabs-mode: nil; -*-
+ </section>
+ <section xml:id="dbdoclet.50438215_marker-1297700">
+ <title>34.6 Example Using the <literal>llapi</literal> Library</title>
+ <para>Use <literal>llapi_file_create</literal> to set Lustre properties for a new file. For a synopsis and description of llapi_file_create and examples of how to use it, see <xref linkend="configurationfilesmoduleparameters"/>.</para>
+ <para>You can set striping from inside programs like <literal>ioctl</literal>. To compile the sample program, you need to download libtest.c and liblustreapi.c files from the Lustre source tree.</para>
+ <para><emphasis role="bold">A simple C program to demonstrate striping API - libtest.c</emphasis></para>
+ <screen>/* -*- mode: c; c-basic-offset: 8; indent-tabs-mode: nil; -*-
* vim:expandtab:shiftwidth=8:tabstop=8:
*
* lustredemo - simple code examples of liblustreapi functions
#include <lustre/liblustreapi.h>
#include <lustre/lustre_user.h>
#define MAX_OSTS 1024
-#define LOV_EA_SIZE(lum, num) (sizeof(*lum) + num * sizeof(*lum->lmm_objects\
-))
+#define LOV_EA_SIZE(lum, num) (sizeof(*lum) + num * sizeof(*lum->lmm_objects))
#define LOV_EA_MAX(lum) LOV_EA_SIZE(lum, MAX_OSTS)
+
+
+
+
/*
This program provides crude examples of using the liblustre API functions
*/
/* Change these definitions to suit */
-
-#define TESTDIR "/tmp" /* R\
-esults directory */
-#define TESTFILE "lustre_dummy" \
- /* Name for the file we create/destroy */
-#define FILESIZE 262144 \
-/* Size of the file in words */
-#define DUMWORD "DEADBEEF" \
- /* Dummy word used to fill files */
-#define MY_STRIPE_WIDTH 2 \
-/* Set this to the number of OST required */
+
+
+
+
+#define TESTDIR "/tmp" /* Results directory */
+#define TESTFILE "lustre_dummy" /* Name for the file we create/destroy */
+#define FILESIZE 262144 /* Size of the file in words */
+#define DUMWORD "DEADBEEF" /* Dummy word used to fill files */
+#define MY_STRIPE_WIDTH 2 /* Set this to the number of OST required */
#define MY_LUSTRE_DIR "/mnt/lustre/ftest"
int close_file(int fd)
{
if (close(fd) < 0) {
- fprintf(stderr, "File close failed: %d (%s)\n", errno, strerror(er\
-rno));
+ fprintf(stderr, "File close failed: %d (%s)\n", errno, strerror(errno));
return -1;
}
return 0;
int open_stripe_file()
{
char *tfile = TESTFILE;
- int stripe_size = 65536; \
- /* System default is 4M */
- int stripe_offset = -1; \
- /* Start at default */
- int stripe_count = MY_STRIPE_WIDTH; \
- /*Single stripe for this demo*/
- int stripe_pattern = 0; \
- /* only RAID 0 at this time */
+ int stripe_size = 65536; /* System default is 4M */
+ int stripe_offset = -1; /* Start at default */
+ int stripe_count = MY_STRIPE_WIDTH; /*Single stripe for this demo*/
+ int stripe_pattern = 0; /* only RAID 0 at this time */
int rc, fd;
/*
*/
We borrow an error message from sanity.c
*/
if (rc) {
- fprintf(stderr,"llapi_file_create failed: %d (%s) \n", rc, st\
-rerror(-rc));
+ fprintf(stderr,"llapi_file_create failed: %d (%s) \n", rc, strerror(-rc));
return -1;
}
/* llapi_file_create closes the file descriptor, we must re-open */
fd = open(tfile, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE, 0644);
if (fd < 0) {
- fprintf(stderr, "Can't open %s file: %d (%s)\n", tfile, errno\
-, strerror(errno));
+ fprintf(stderr, "Can't open %s file: %d (%s)\n", tfile, errno, strerror(errno));
return -1;
}
return fd;
/* output a list of uuids for this file */
int get_my_uuids(int fd)
{
- struct obd_uuid uuids[1024], *uuidp; \
- /* Output var */
+ struct obd_uuid uuids[1024], *uuidp; /* Output var */
int obdcount = 1024;
int rc,i;
rc = llapi_lov_get_uuids(fd, uuids, &obdcount);
if (rc != 0) {
- fprintf(stderr, "get uuids failed: %d (%s)\n",errno, strerror(errn\
-o));
+ fprintf(stderr, "get uuids failed: %d (%s)\n",errno, strerror(errno));
}
printf("This file system has %d obds\n", obdcount);
for (i = 0, uuidp = uuids; i < obdcount; i++, uuidp++) {
rc = llapi_file_get_stripe(path, lump);
if (rc != 0) {
- fprintf(stderr, "get_stripe failed: %d (%s)\n",errno, strerror(err\
-no));
+ fprintf(stderr, "get_stripe failed: %d (%s)\n",errno, strerror(errno));
return -1;
}
printf("Lov stripe count %hu\n", lump->lmm_stripe_count);
printf("Lov stripe offset %u\n", lump->lmm_stripe_offset);
for (i = 0; i < lump->lmm_stripe_count; i++) {
- printf("Object index %d Objid %llu\n", lump->lmm_objects[i].l_ost_i\
-dx, lump->lmm_objects[i].l_object_id);
+ printf("Object index %d Objid %llu\n", lump->lmm_objects[i].l_ost_idx, lump->lmm_objects[i].l_object_id);
}
exit(rc);
}
</screen>
-
<para><emphasis role="bold">Makefile for sample application:</emphasis></para>
- <screen>
+ <para><emphasis role="bold">Makefile for sample application:</emphasis></para>
+ <screen>
gcc -g -O2 -Wall -o lustredemo libtest.c -llustreapi
clean:
rm -f core lustredemo *.o
rm -f /mnt/lustre/ftest/lustre_dummy
cp lustredemo /mnt/lustre/ftest/
</screen>
- <section remap="h5">
- <title>See Also</title>
- <para>
- <xref linkend="dbdoclet.50438215_30970"/>llapi_file_create,
- <xref linkend="dbdoclet.50438215_50149"/>llapi_file_get_stripe,
- <xref linkend="dbdoclet.50438215_86607"/>llapi_file_open,
- <xref linkend="dbdoclet.50438215_12433"/>llapi_quotactl</para>
- </section>
+ <section remap="h5">
+ <title>See Also</title>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438215_30970"/>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438215_50149"/>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438215_86607"/>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <xref linkend="dbdoclet.50438215_12433"/>
+ </para>
+ </listitem>
+ </itemizedlist>
</section>
+ </section>
</chapter>
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandinglustre">
- <info>
- <title xml:id="understandinglustre.title">Understanding Lustre</title>
- </info>
+ <title xml:id="understandinglustre.title">Understanding Lustre</title>
+ <info/>
<para>This chapter describes the Lustre architecture and features of Lustre. It includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="understandinglustre.whatislustre">
- <title>1.1 <anchor xml:id="dbdoclet.50438250_92658" xreflabel=""/>What Lustre Is (and What It Isn't)</title>
+ <title>What Lustre Is (and What It Isn't)</title>
<para>Lustre is a storage architecture for clusters. The central component of the Lustre architecture is the Lustre file system, which is supported on the Linux operating system and provides a POSIX-compliant UNIX file system interface.</para>
<para>The Lustre storage architecture is used for many different kinds of clusters. It is best known for powering seven of the ten largest high-performance computing (HPC) clusters worldwide, with tens of thousands of client systems, petabytes (PB) of storage and hundreds of gigabytes per second (GB/sec) of I/O throughput. Many HPC sites use Lustre as a site-wide global file system, serving dozens of clusters.</para>
<para>The ability of a Lustre file system to scale capacity and performance for any need reduces the need to deploy many separate file systems, such as one for each compute cluster. Storage management is simplified by avoiding the need to copy data between compute clusters. In addition to aggregating storage capacity of many servers, the I/O throughput is also aggregated and scales with additional servers. Moreover, throughput and/or capacity can be easily increased by adding servers dynamically.</para>
<para>While Lustre can function in many work environments, it is not necessarily the best choice for all applications. It is best suited for uses that exceed the capacity that a single server can provide, though in some use cases Lustre can perform better with a single server than other filesystems due to its strong locking and data coherency.</para>
<para>Lustre is currently not particularly well suited for "peer-to-peer" usage models where there are clients and servers running on the same node, each sharing a small amount of storage, due to the lack of Lustre-level data replication. In such uses, if one client/server fails, then the data stored on that node will not be accessible until the node is restarted.</para>
<section remap="h3">
- <title>1.1.1 Lustre <anchor xml:id="dbdoclet.50438250_marker-1293792" xreflabel=""/>Features</title>
+ <title>Lustre Features</title>
<para>Lustre runs on a variety of vendor's kernels. For more details, see <link xl:href="http://wiki.whamcloud.com/">Lustre Release Information</link> on the Whamcloud wiki.</para>
<para>A Lustre installation can be scaled up or down with respect to the number of client nodes, disk storage and bandwidth. Scalability and performance are dependent on available disk and network bandwith and the processing power of the servers in the system. Lustre can be deployed in a wide variety of configurations that can be scaled well beyond the size and performance observed in production systems to date.</para>
<para><xref linkend="understandinglustre.tab1"/> shows the practical range of scalability and performance characteristics of the Lustre file system and some test results in production systems.</para>
<thead>
<row>
<entry>
- <para><emphasis role="bold">
-
- </emphasis></para>
+ <para><emphasis role="bold"/></para>
</entry>
<entry>
<para><emphasis role="bold">Feature</emphasis></para>
<para>Other Lustre features are:</para>
<itemizedlist>
<listitem>
- <para><emphasis role="bold">Performance-enhanced ext4 file system:</emphasis> Lustre uses a modified version of the ext4 journaling file system to store data and metadata. This version, called <emphasis role="italic">ldiskfs</emphasis>, has been enhanced to improve performance and provide additional functionality needed by Lustre.</para>
+ <para><emphasis role="bold">Performance-enhanced ext4 file system:</emphasis> Lustre uses a modified version of the ext4 journaling file system to store data and metadata. This version, called <emphasis role="italic">
+ <literal>ldiskfs</literal>
+ </emphasis>, has been enhanced to improve performance and provide additional functionality needed by Lustre.</para>
</listitem>
<listitem>
<para><emphasis role="bold">POSIX compliance</emphasis> : The full POSIX test suite passes with limited exceptions on Lustre clients. In a cluster, most operations are atomic so that clients never see stale data or metadata. Lustre supports mmap() file I/O.</para>
</section>
</section>
<section xml:id="understandinglustre.components">
- <title>1.2 <anchor xml:id="dbdoclet.50438250_17402" xreflabel=""/>Lustre Components</title>
+ <title>Lustre Components</title>
<para>An installation of the Lustre software includes a management server (MGS) and one or more Lustre file systems interconnected with Lustre networking (LNET).</para>
<para>A basic configuration of Lustre components is shown in <xref linkend="understandinglustre.fig.cluster"/>.</para>
<figure>
</mediaobject>
</figure>
<section remap="h3">
- <title>1.2.1 Management Server (MGS)</title>
+ <title>Management Server (MGS)</title>
<para>The MGS stores configuration information for all the Lustre file systems in a cluster and provides this information to other Lustre components. Each Lustre target contacts the MGS to provide information, and Lustre clients contact the MGS to retrieve information.</para>
<para>It is preferable that the MGS have its own storage space so that it can be managed independently. However, the MGS can be co-located and share storage space with an MDS as shown in <xref linkend="understandinglustre.fig.cluster"/>.</para>
</section>
<section remap="h3">
- <title>1.2.2 Lustre File System Components</title>
+ <title>Lustre File System Components</title>
<para>Each Lustre file system consists of the following components:</para>
<itemizedlist>
<listitem>
<thead>
<row>
<entry>
- <para><emphasis role="bold">
-
- </emphasis></para>
+ <para><emphasis role="bold"/></para>
</entry>
<entry>
<para><emphasis role="bold">Required attached storage</emphasis></para>
<para>For additional hardware requirements and considerations, see <xref linkend="settinguplustresystem"/>.</para>
</section>
<section remap="h3">
- <title>1.2.3 Lustre Networking (LNET)</title>
+ <title>Lustre Networking (LNET)</title>
<para>Lustre Networking (LNET) is a custom networking API that provides the communication infrastructure that handles metadata and file I/O data for the Lustre file system servers and clients. For more information about LNET, see something <xref linkend="understandinglustrenetworking"/>.</para>
</section>
<section remap="h3">
- <title>1.2.4 Lustre Cluster</title>
+ <title>Lustre Cluster</title>
<para>At scale, the Lustre cluster can include hundreds of OSSs and thousands of clients (see <xref linkend="understandinglustre.fig.lustrescale"/>). More than one type of network can be used in a Lustre cluster. Shared storage between OSSs enables failover capability. For more details about OSS failover, see <xref linkend="understandingfailover"/>.</para>
<figure>
<title xml:id="understandinglustre.fig.lustrescale">Lustre cluster at scale</title>
</section>
</section>
<section xml:id="understandinglustre.storageio">
- <title>1.3 <anchor xml:id="dbdoclet.50438250_38077" xreflabel=""/>Lustre Storage and I/O</title>
+ <title>Lustre Storage and I/O</title>
<para>In a Lustre file system, a file stored on the MDT points to one or more objects associated with a data file, as shown in <xref linkend="understandinglustre.fig.mdtost"/>. Each object contains data and is stored on an OST. If the MDT file points to one object, all the file data is stored in that object. If the file points to more than one object, the file data is 'striped' across the objects (using RAID 0) and each object is stored on a different OST. (For more information about how striping is implemented in Lustre, see <xref linkend="dbdoclet.50438250_89922"/>)</para>
<para>In <xref linkend="understandinglustre.fig.mdtost"/>, each filename points to an inode. The inode contains all of the file attributes, such as owner, access permissions, Lustre striping layout, access time, and access control. Multiple filenames may point to the same inode.</para>
<figure>
<para>The available bandwidth of a Lustre file system is determined as follows:</para>
<itemizedlist>
<listitem>
- <para> The <emphasis>network bandwidth</emphasis> equals the aggregated bandwidth of the OSSs to the targets.</para>
+ <para>The <emphasis>network bandwidth</emphasis> equals the aggregated bandwidth of the OSSs to the targets.</para>
</listitem>
<listitem>
- <para> The <emphasis>disk bandwidth</emphasis> equals the sum of the disk bandwidths of the storage targets (OSTs) up to the limit of the network bandwidth.</para>
+ <para>The <emphasis>disk bandwidth</emphasis> equals the sum of the disk bandwidths of the storage targets (OSTs) up to the limit of the network bandwidth.</para>
</listitem>
<listitem>
- <para> The <emphasis>aggregate bandwidth</emphasis> equals the minimium of the disk bandwidth and the network bandwidth.</para>
+ <para>The <emphasis>aggregate bandwidth</emphasis> equals the minimium of the disk bandwidth and the network bandwidth.</para>
</listitem>
<listitem>
- <para> The <emphasis>available file system space</emphasis> equals the sum of the available space of all the OSTs.</para>
+ <para>The <emphasis>available file system space</emphasis> equals the sum of the available space of all the OSTs.</para>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438250_89922">
<imagedata fileref="./figures/File_Striping.png"/>
</imageobject>
<textobject>
- <phrase> File striping pattern across three OSTs for three different data files. The file is sparse and missing chunk 6. </phrase>
+ <phrase>File striping pattern across three OSTs for three different data files. The file is sparse and missing chunk 6. </phrase>
</textobject>
</mediaobject>
</figure>
<?xml version='1.0' encoding='UTF-8'?>
-<!-- This document was created with Syntext Serna Free. -->
-<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandinglustrenetworking">
- <info>
- <title xml:id="understandinglustrenetworking.title">Understanding Lustre Networking (LNET)</title>
- </info>
+<!-- This document was created with Syntext Serna Free. --><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="understandinglustrenetworking">
+ <title xml:id="understandinglustrenetworking.title">Understanding Lustre Networking (LNET)</title>
<para>This chapter introduces Lustre Networking (LNET) and includes the following sections:</para>
<itemizedlist>
<listitem>
</listitem>
</itemizedlist>
<section xml:id="dbdoclet.50438191_22878">
- <title>2.1 Introducing LNET</title>
+ <title>Introducing LNET</title>
<para>In a cluster with a Lustre file system, the system network connecting the servers and the clients is implemented using Lustre Networking (LNET), which provides the communication infrastructure required by the Lustre file system.</para>
<para>LNET supports many commonly-used network types, such as InfiniBand and IP networks, and allows simultaneous availability across multiple network types with routing between them. Remote Direct Memory Access (RDMA) is permitted when supported by underlying networks using the appropriate Lustre network driver (LND). High availability and recovery features enable transparent recovery in conjunction with failover servers.</para>
<para>An LND is a pluggable driver that provides support for a particular network type. LNDs are loaded into the driver stack, with one LND for each network type in use.</para>
<para>For information about administering LNET, see <xref linkend="adminlustrepart3"/>.</para>
</section>
<section xml:id="dbdoclet.50438191_19625">
- <title>2.2 Key Features of LNET</title>
+ <title>Key Features of LNET</title>
<para>Key features of LNET include:</para>
<itemizedlist>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_57931" xreflabel=""/>RDMA, when supported by underlying networks such as InfiniBand or Myrinet MX</para>
+ <para>RDMA, when supported by underlying networks such as InfiniBand or Myrinet MX</para>
</listitem>
<listitem>
<para>Support for many commonly-used network types such as InfiniBand and TCP/IP</para>
<para>Lustre can use bonded networks, such as bonded Ethernet networks, when the underlying network technology supports bonding. For more information, see <xref linkend="settingupbonding"/>.</para>
</section>
<section xml:id="dbdoclet.50438191_20721">
- <title>2.3 Supported Network Types</title>
+ <title>Supported Network Types</title>
<para>LNET includes LNDs to support many network types including:</para>
<itemizedlist>
<listitem>
<para> Myrinet: MX</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_marker-1289887" xreflabel=""/> RapidArray: ra</para>
+ <para> RapidArray: ra</para>
</listitem>
<listitem>
- <para><anchor xml:id="dbdoclet.50438191_marker-1289889" xreflabel=""/> Quadrics: Elan</para>
+ <para> Quadrics: Elan</para>
</listitem>
</itemizedlist>
</section>
git://git.whamcloud.com / doc / manual.git / commitdiff