2 .TH lnetctl 8 "2017 Jan 12" Lustre "configuration utilities"
8 .B lnetctl --list-commands
10 \fBlnetctl\fR \fB<cmd> <subcmd> [optional parameters]\fR
14 \fBlnetctl\fR is used to configure LNet parameters allowing various configuration
15 and debugging features to be accessed\.
18 \fBlnetctl\fR can be invoked in interactive mode by issuing lnetctl command\.
19 After that, commands are issued as below\.
22 To get a complete listing of available commands, type
24 at the lnetctl prompt\. To get basic help on the meaning and syntax of a command
25 type \fB<command>\fR \-\-help\.
28 For non\-interactive use the \fBlnetctl\fR utility can be invoked from the
29 command line as follows:
32 lnetctl \fIcommand\fR \fIsubcommand\fR [optional parameters]
34 .SS "LNet Initialization:"
37 \fBlnetctl lnet\fR configure [\-\-all]
38 Brings up the LNet Network Interface (NI) system\. If the \-\-all option is
39 provided it will load network interfaces defined in the modprobe files\.
40 Otherwise, it doesn\'t initialize any networks by default\.
43 \fBlnetctl lnet\fR unconfigure
44 Brings down the LNet Network Interface (NI) system including any configured
47 .SS "Network Configuration"
51 Configures a network interface either given the network name and physical
52 interface device name, or given the ip2net parameter\. Other parameters
56 \-\-net: net name (e.g. tcp0)
59 \-\-if: physical interface (e.g. eth0)
62 \-\-ip2net: specify networks based on IP address patterns
65 \-\-peer\-timeout: time to wait before declaring a peer dead (in seconds).
66 Default value for o2iblnd and socklnd is 180 seconds.
69 \-\-peer\-credits: define the max number of in\-flight messages per peer.
72 \-\-peer\-buffer\-credits: the max number of routed in\-flight messages
76 \-\-credits: The total number of in\-flight messages over a network interface.
79 \-\-cpt: The CPU partitions on which the created network interface is bound to.
80 Refer to the Lustre Manual Section "Binding Network Interface Against CPU
81 Partitions" for more details. For example to bind a Network Interface to
82 CPU partitions 0 and 1, you would specify this parameter as \-\-cpt [0,
90 Delete a network interface given the network name\.
93 \-\-net: net name (e.g. tcp0)
99 \fBlnetctl net\fR show
100 Show all currently configured network interfaces if no parameters given or filter
101 on the network name\. More details can be shown by specifying the \-\-verbose
105 \-\-net: net name (e.g. tcp0) to filter on
108 \-\-verbose: display detailed output per network
111 .SS "Peer Configuration"
113 \fBlnetctl peer\fR add
114 Configure an LNET peer with at least one supplied NID\. The primary NID must be specified. By default, peers are marked as multi-rail capable\.
118 \-\-nid: one or more peer NIDs to add to the peer\.
122 \-\-prim_nid: Primary NID of the peer\.
125 \-\-non_mr: create this peer as not Multi-Rail capable\.
130 \fBlnetctl peer\fR del
131 Delete a peer NID. The primary NID must be specified. If the removed NID is the primary NID, the peer entry will be deleted.
135 \-\-nid: one or more peer NIDs to remove from the peer\.
139 \-\-prim_nid: Primary NID of the peer\.
144 \fBlnetctl peer\fR show
145 Show configured peers. By default, lists all peers and associated NIDs.
149 \-\-nid: list of primary nids to filter on
153 \-\-verbose: Include extended statistics, including credits and counters.
158 .SS "Route Configuration"
161 \fBlnetctl route\fR add
165 \-\-net: net name (e.g. tcp0)
168 \-\-gateway: gateway nid (e.g. 10\.1\.1\.2@tcp)
171 \-\-hop: number to final destination (1 < hops < 255)
174 \-\-priority: priority of route (0 \- highest prio)
180 \fBlnetctl route\fR del
181 Delete a route specified via the network and gateway\.
184 \-\-net: net name (e.g. tcp0)
187 \-\-gateway: gateway nid (e.g. 10\.1\.1\.2@tcp)
193 \fBlnetctl route\fR show
194 Show all currently configured routes if no parameters given, or filter on
195 given parameters\. More details can be shown by specifying the \-\-verbose
199 \-\-net: net name (e.g. tcp0) to filter on
202 \-\-gateway: gateway nid (e.g. 10\.1\.1\.2@tcp) to filter on
205 \-\-hop: number to final destination (1 < hops < 255) to filter on
208 \-\-priority: priority of route (0 \- highest prio to filter on)
211 \-\-verbose: display detailed output per route
216 .SS "Routing Information"
219 \fBlnetctl routing\fR show
220 Show router buffers values as well as show the status of routing (IE: whether
221 the node is set to be a router)
224 Individual values can be set using the \fBlnetctl set\fR command\.
227 \fBlnetctl set\fR tiny_buffers \fIvalue\fR
228 Set the number of tiny buffers in the system\. This is the total number of tiny
229 buffers for all CPU partitions\.
232 \fBlnetctl set\fR small_buffers \fIvalue\fR
233 Set the number of small buffers in the system\. This is the total number of
234 small buffers for all CPU partitions\.
237 \fBlnetctl set\fR large_buffers \fIvalue\fR
238 Set the number of large buffers in the system\. This is the total number of
239 large buffers for all CPU partitions\.
242 \fBlnetctl set\fR routing \fI[0, 1]\fR
243 0 value indicates to disable routing\. 1 value indicates to enable routing\.
244 When routing is disabled the values of the buffers that might have been changed
245 are not remembered, and the next time routing is enabled the default buffer
249 \fBlnetctl set\fR drop_asym_route \fI[0, 1]\fR
250 0 value indicates to accept asymmetrical route messages\. 1 value indicates to
251 drop them\. Asymmetrical route is when a message from a remote peer is coming
252 through a router that would not be used by this node to reach the remote peer\.
255 \fBlnetctl set\fR response_tracking \fI[0, 1, 2, 3]\fR
256 Set the behavior of response tracking\.
257 0 - Only LNet pings and discovery pushes utilize response tracking\.
258 1 - GETs are eligible for response tracking\.
259 2 - PUTs are eligible for response tracking\.
260 3 - Both PUTs and GETs are eligible for response tracking (default)\.
261 Note: Regardless of the value of the response_tracking parameter LNet
262 pings and discovery pushes always utilize response tracking\.
264 .SS "Import and Export YAML Configuration Files"
265 LNet configuration can be represented in YAML format\. A YAML configuration
266 file can be passed to the lnetctl utility via the \fBimport\fR command\. The
267 lnetctl utility will attempt to configure all elements defined in the YAML
271 Similarly the \fBexport\fR command can be used to dump all supported LNet
272 configuration to stdout\. The output can be redirected to a file\.
275 \fBlnetctl import\fR \fIFILE\fR:
279 \fBlnetctl import\fR < \fIFILE\fR
280 \fBimport\fR command uses the specified YAML configuration file to configure
281 LNet parameters defined within\. The import command by default adds the LNet
282 parameters defined in the YAML file, but this default behavior can be
283 overwritten by specifying the desired behavior\.
286 \-\-add: add configuration
289 \-\-del: delete configuration
292 \-\-show: show configuration
295 \-\-help: display this help
298 \fBlnetctl export\fR \fIFILE\fR:
302 \fBlnetctl export\fR > \fIFILE\fR
303 \fBexport\fR command dumps the supported LNet configuration in YAML format to
304 stdout, which can be redirected to a normal file\. The output of the
305 \fBimport\fR command can be used as input to the \fBexport\fR command\.
308 \-\-help: display this help
310 .SS "LNet Statistics"
317 \-> Number of messages allocated
320 \-> Maximum number of messages allocated
323 \-> Number of errors encountered
326 \-> Number of messages sent
329 \-> Number of messages received
332 \-> Number of messages routed
335 \-> Total size in bytes of messages sent
338 \-> Total size in bytes of messages received
341 \-> Total size in bytes of messages routed
344 \-> Total size in bytes of messages dropped
349 .SS "Showing Peer Credits"
352 \fBlnetctl peer_credits\fR
353 Show details on configured peer credits
362 \-> Reference count on the peer
365 \-> Maximum transmit credits
368 \-> Available transmit credits
371 \-> Available router credits
374 \-> Minimum router credits\.
379 Output a list of the commands supported by the lnetctl utility
382 .SS "Initializing LNet after load"
385 lnetctl lnet configure
388 lnetctl lnet configure \-\-all
392 .SS "Shutting down LNet"
395 lnetctl lnet unconfigure
402 lnetctl net add \-\-net tcp0 \-\-if eth0
405 lnetctl net add \-\-ip2net "tcp0(eth0) 192\.168\.0\.[2,4]; tcp0 192\.168\.0\.*;
406 o2ib0 132\.6\.[1\-3]\.[2\-8/2]"
413 lnetctl net del \-\-net tcp0
420 lnetctl net show \-\-verbose:
442 peer_buffer_credits: 0
448 \- nid: 192\.168\.205\.130@tcp1
472 peer_buffer_credits: 0
482 lnetctl route add \-\-net tcp0 \-\-gateway 10\.10\.10\.1@tcp1 \-\-hop 1
490 lnetctl route del \-\-net tcp0 \-\-gateway 10\.10\.10\.1@tcp1
497 lnetctl route show \-\-verbose
508 gateway: 192\.168\.205\.131@tcp1
514 priority: 0 state: down
579 .SS "Setting variables"
582 lnetctl set tiny_buffers 2048
585 lnetctl set small_buffers 16384
588 lnetctl set large_buffers 256
591 lnetctl set routing 1
595 .SS "Importing YAML files for configuring"
598 lnetctl import lnet\.conf
601 lnetctl import < lnet\.conf
605 .SS "Exporting LNet Configuration"
608 lnetctl export lnet\.conf
611 lnetctl export > lnet\.conf
615 .SS "Showing LNet Stats"
660 .SS "Showing peer information"
671 \- primary nid: 10\.148\.0\.8@o2ib
680 \- nid: 10\.148\.0\.8@o2ib
686 \- primary nid: 10\.148\.0\.20@o2ib
695 \- nid: 10\.148\.0\.20@o2ib
701 \- nid: 10\.148\.0\.25@o2ib