1 <?xml version='1.0' encoding='UTF-8'?><chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US" xml:id="lnetconfigurationapi">
2 <title xml:id="lnetconfigurationapi.title">LNet Configuration C-API</title>
3 <para>This section describes the LNet Configuration C-API library. This
4 API allows the developer to programatically configure LNet. It provides
5 APIs to add, delete and show LNet configuration items listed below. The
6 API utilizes IOCTL to communicate with the kernel. Changes take effect
7 immediately and do not require restarting LNet. API calls are
12 <para>Configuring LNet</para>
15 <para>Enabling/Disabling routing</para>
18 <para>Adding/removing/showing Routes</para>
21 <para>Adding/removing/showing Networks</para>
24 <para>Configuring Router Buffer Pools</para>
30 <primary>LNet</primary>
31 <secondary>capi general information</secondary>
32 </indexterm>General API Information</title>
36 <primary>LNet</primary>
37 <secondary>capi return code</secondary>
38 </indexterm>API Return Code</title>
39 <screen>LUSTRE_CFG_RC_NO_ERR 0
40 LUSTRE_CFG_RC_BAD_PARAM -1
41 LUSTRE_CFG_RC_MISSING_PARAM -2
42 LUSTRE_CFG_RC_OUT_OF_RANGE_PARAM -3
43 LUSTRE_CFG_RC_OUT_OF_MEM -4
44 LUSTRE_CFG_RC_GENERIC_ERR -5</screen>
48 <primary>LNet</primary>
49 <secondary>capi input params</secondary>
50 </indexterm>API Common Input Parameters</title>
51 <para>All APIs take as input a sequence number. This is a number
52 that's assigned by the caller of the API, and is returned in the
53 YAML error return block. It is used to associate the request with
54 the response. It is especially useful when configuring via the
55 YAML interface, since typically the YAML interface is used to
56 configure multiple items. In the
57 return Error block, it is desired to know which items were
58 configured properly and which were not configured properly. The
59 sequence number achieves this purpose.</para>
63 <primary>LNet</primary>
64 <secondary>capi output params</secondary>
65 </indexterm>API Common Output Parameters</title>
69 <primary>LNet</primary>
70 <secondary>cyaml</secondary>
71 </indexterm>Internal YAML Representation (cYAML)</title>
72 <para>Once a YAML block is parsed it needs to be stored
73 structurally in order to facilitate passing it to different
74 functions, querying it and printing it. Also it is required to
75 be able to build this internal representation from data returned
76 from the kernel and return it to the caller, which can query and
77 print it. This structure
78 representation is used for the Error and Show API Out
79 parameters. For this YAML is internally represented via this
81 <screen>typedef enum {
82 EN_YAML_TYPE_FALSE = 0,
89 } cYAML_object_type_t;
91 typedef struct cYAML {
92 /* next/prev allow you to walk array/object chains. */
93 struct cYAML *cy_next, *cy_prev;
94 /* An array or object item will have a child pointer pointing
95 to a chain of the items in the array/object. */
96 struct cYAML *cy_child;
97 /* The type of the item, as above. */
98 cYAML_object_type_t cy_type;
99 /* The item's string, if type==EN_YAML_TYPE_STRING */
100 char *cy_valuestring;
101 /* The item's number, if type==EN_YAML_TYPE_NUMBER */
103 /* The item's number, if type==EN_YAML_TYPE_NUMBER */
104 double cy_valuedouble;
105 /* The item's name string, if this item is the child of,
106 or is in the list of subitems of an object. */
108 /* user data which might need to be tracked per object */
114 <primary>LNet</primary>
115 <secondary>error block</secondary>
116 </indexterm>Error Block</title>
117 <para>All APIs return a cYAML error block. This error block has
118 the following format, when it's printed out. All configuration
119 errors shall be represented in a YAML sequence</para>
122 errno: <error number>
123 seqno: <sequence number>
124 descr: <error description>
131 descr: Missing mandatory parameter(s) - network</screen>
135 <primary>LNet</primary>
136 <secondary>show block</secondary>
137 </indexterm>Show Block</title>
138 <para>All Show APIs return a cYAML show block. This show block
139 represents the information requested in YAML format. Each
140 configuration item has its own YAML syntax. The YAML syntax of
141 all supported configuration items is described later in this
142 document. Below is an example of a show block:</para>
144 - nid: 192.168.206.130@tcp4
151 peer_buffer_credits: 30
158 <primary>LNet</primary>
159 <secondary>show block</secondary>
160 </indexterm>The LNet Configuration C-API</title>
164 <primary>LNet</primary>
165 <secondary>lustre_lnet_config_ni_system</secondary>
166 </indexterm>Configuring LNet</title>
170 * lustre_lnet_config_ni_system
171 * Initialize/Uninitialize the LNet NI system.
173 * up - whether to init or uninit the system
174 * load_ni_from_mod - load NI from mod params.
175 * seq_no - sequence number of the request
176 * err_rc - [OUT] struct cYAML tree describing the error. Freed by
179 int lustre_lnet_config_ni_system(bool up, bool load_ni_from_mod,
180 int seq_no, struct cYAML **err_rc);</screen>
181 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
182 <para>IOC_LIBCFS_CONFIGURE or IOC_LIBCFS_UNCONFIGURE</para>
183 <para><emphasis role="bold">Description:</emphasis></para>
184 <para><emphasis role="bold">Configuring LNet</emphasis>
186 <para>Initialize LNet internals and load any networks specified in the module
187 parameter if <literal>load_ni_from_mod</literal> is set. Otherwise do not
188 load any network interfaces.</para>
189 <para><emphasis role="bold">Unconfiguring LNet</emphasis></para>
190 <para>Bring down LNet and clean up network itnerfaces, routes and all LNet
192 <para><emphasis role="bold">Return Value</emphasis></para>
193 <para>0: if success</para>
194 <para>-errno: if failure</para>
198 <primary>LNet</primary>
199 <secondary>lustre_lnet_enable_routing</secondary>
200 </indexterm>Enabling and Disabling Routing</title>
203 * lustre_lnet_enable_routing
204 * Send down an IOCTL to enable or disable routing
206 * enable - 1 to enable routing, 0 to disable routing
207 * seq_no - sequence number of the request
208 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
210 extern int lustre_lnet_enable_routing(int enable,
212 cYAML **err_rc);</screen>
213 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
214 <para>IOC_LIBCFS_ENABLE_RTR </para>
215 <para><emphasis role="bold">Description:</emphasis></para>
216 <para><emphasis role="bold">Enabling Routing</emphasis>
218 <para>The router buffer pools are allocated using the default values. Internally the node
219 is then flagged as a Router node. The node can be used as a router from this
221 <para><emphasis role="bold">Disabling Routing</emphasis></para>
222 <para>The unused router buffer pools are freed. Buffers currently
223 in use are not freed until they are returned to the unused list.
224 Internally the node routing flag is turned off. Any subsequent
225 messages not destined to this node are dropped. </para>
226 <para><emphasis role="bold">Enabling Routing on an already enabled
227 node, or vice versa</emphasis></para>
228 <para>In both these cases the LNet Kernel module ignores this request. </para>
229 <para><emphasis role="bold">Return Value</emphasis></para>
230 <para>-ENOMEM: if there is no memory to allocate buffer pools</para>
231 <para>0: if success</para>
235 <primary>LNet</primary>
236 <secondary>lustre_lnet_config_route</secondary>
237 </indexterm>Adding Routes</title>
240 * lustre_lnet_config_route
241 * Send down an IOCTL to the kernel to configure the route
245 * hops - number of hops passed down by the user
246 * prio - priority of the route
247 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
249 extern int lustre_lnet_config_route(char *nw, char *gw,
252 cYAML **err_rc);</screen>
253 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
254 <para>IOC_LIBCFS_ADD_ROUTE</para>
255 <para><emphasis role="bold">Description:</emphasis></para>
256 <para>The LNet Kernel module adds this route to the list of
257 existing routes, if one doesn't already exist. If hop parameter is
258 not specified (IE: -1) then the hop count is set to 1. If the
259 priority parameter is not specified (IE: -1) then the priority is
260 set to 0. All routes with the same hop and priority are used in
261 round robin. Routes with lower number of hops and/or higher
262 priority are preferred. 0 is the highest priority.</para>
263 <para>If a route already exists the request to add the same route is ignored.</para>
264 <para><emphasis role="bold">Return Value</emphasis></para>
265 <para>-EINVAL: if the network of the route is local</para>
266 <para>-ENOMEM: if there is no memory</para>
267 <para>-EHOSTUNREACH: if the host is not on a local network</para>
268 <para>0: if success</para>
272 <primary>LNet</primary>
273 <secondary>lustre_lnet_del_route</secondary>
274 </indexterm>Deleting Routes</title>
277 * lustre_lnet_del_route
278 * Send down an IOCTL to the kernel to delete a route
283 extern int lustre_lnet_del_route(char *nw, char *gw,
285 cYAML **err_rc);</screen>
286 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
287 <para>IOC_LIBCFS_DEL_ROUTE</para>
288 <para><emphasis role="bold">Description:</emphasis></para>
289 <para>LNet will remove the route which matches the network and gateway passed in. If
290 no route matches, then the operation fails with an appropriate error number.</para>
291 <para><emphasis role="bold">Return Value</emphasis></para>
292 <para>-ENOENT: if the entry being deleted doesn't exist</para>
293 <para>0: if success</para>
297 <primary>LNet</primary>
298 <secondary>lustre_lnet_show_route</secondary>
299 </indexterm>Showing Routes</title>
302 * lustre_lnet_show_route
303 * Send down an IOCTL to the kernel to show routes
304 * This function will get one route at a time and filter according to
305 * provided parameters. If no filter is provided then it will dump all
306 * routes that are in the system.
308 * nw - network. Optional. Used to filter output
309 * gw - gateway. Optional. Used to filter ouptut
310 * hops - number of hops passed down by the user
311 * Optional. Used to filter output.
312 * prio - priority of the route. Optional. Used to filter output.
313 * detail - flag to indicate whether detail output is required
314 * show_rc - [OUT] The show output in YAML. Must be freed by caller.
315 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
317 extern int lustre_lnet_show_route(char *nw, char *gw,
318 int hops, int prio, int detail,
321 cYAML **err_rc);</screen>
322 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
323 <para>IOC_LIBCFS_GET_ROUTE</para>
324 <para><emphasis role="bold">Description:</emphasis></para>
325 <para>The routes are fetched from the kernel one by one and packed
326 in a cYAML block, after filtering according to the parameters
327 passed in. The cYAML block is then returned to the caller of the
329 <para>An example with the detail parameter set to 1</para>
332 gateway: 192.168.205.130@tcp
336 <para>An Example with the detail parameter set to 0</para>
339 gateway: 192.168.205.130@tcp</screen>
340 <para><emphasis role="bold">Return Value</emphasis></para>
341 <para>-ENOMEM: If no memory</para>
342 <para>0: if success</para>
346 <primary>LNet</primary>
347 <secondary>lustre_lnet_config_net</secondary>
348 </indexterm>Adding a Network Interface</title>
351 * lustre_lnet_config_net
352 * Send down an IOCTL to configure a network.
354 * net - the network name
355 * intf - the interface of the network of the form net_name(intf)
356 * peer_to - peer timeout
357 * peer_cr - peer credit
358 * peer_buf_cr - peer buffer credits
359 * - the above are LND tunable parameters and are optional
360 * credits - network interface credits
362 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
364 extern int lustre_lnet_config_net(char *net,
372 cYAML **err_rc);</screen>
373 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
374 <para>IOC_LIBCFS_ADD_NET</para>
375 <para><emphasis role="bold">Description:</emphasis></para>
376 <para>A new network is added and initialized. This has the same
377 effect as configuring a network from the module parameters. The
378 API allows the specification of network parameters such as the
379 peer timeout, peer credits, peer buffer credits and credits. The
380 CPU affinity of the network interface being added can also be
381 specified. These parameters become
382 network specific under Dynamic LNet Configuration (DLC), as
383 opposed to being per LND as it was previously.</para>
384 <para>If an already existing network is added the request is ignored.</para>
385 <para><emphasis role="bold">Return Value</emphasis></para>
386 <para>-EINVAL: if the network passed in is not recognized.</para>
387 <para>-ENOMEM: if no memory</para>
388 <para>0: success</para>
392 <primary>LNet</primary>
393 <secondary>lustre_lnet_del_net</secondary>
394 </indexterm>Deleting a Network Interface</title>
397 * lustre_lnet_del_net
398 * Send down an IOCTL to delete a network.
400 * nw - network to delete.
401 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
403 extern int lustre_lnet_del_net(char *nw,
405 cYAML **err_rc);</screen>
406 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
407 <para>IOC_LIBCFS_DEL_NET</para>
408 <para><emphasis role="bold">Description:</emphasis></para>
409 <para>The network interface specified is deleted. All resources
410 associated with this network interface are freed. All routes going
411 over that Network Interface are cleaned up.</para>
412 <para>If a non existent network is deleted then the call return -EINVAL.</para>
413 <para><emphasis role="bold">Return Value</emphasis></para>
414 <para>-EINVAL: if the request references a non-existent network.</para>
415 <para>0: success</para>
419 <primary>LNet</primary>
420 <secondary>lustre_lnet_show_net</secondary>
421 </indexterm>Showing Network Interfaces</title>
424 * lustre_lnet_show_net
425 * Send down an IOCTL to show networks.
426 * This function will use the nw paramter to filter the output. If it's
427 * not provided then all networks are listed.
429 * nw - network to show. Optional. Used to filter output.
430 * detail - flag to indicate if we require detail output.
431 * show_rc - [OUT] The show output in YAML. Must be freed by caller.
432 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
434 extern int lustre_lnet_show_net(char *nw, int detail,
437 cYAML **err_rc);</screen>
438 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
439 <para>IOC_LIBCFS_GET_NET</para>
440 <para><emphasis role="bold">Description:</emphasis></para>
441 <para>The network interfaces are queried one at a time from the
442 kernel and packed in a cYAML block, after filtering on the network
443 (EX: tcp). If the detail field is set to 1, then the tunable
444 section of the show block is included in the return.</para>
445 <para>An example of the detailed output</para>
447 nid: 192.168.206.130@tcp4
454 peer_buffer_credits: 30
456 <para>An example of none detailed output</para>
458 nid: 192.168.206.130@tcp4
461 intf-0: eth0</screen>
462 <para><emphasis role="bold">Return Value</emphasis></para>
463 <para>-ENOMEM: if no memory to allocate the error or show blocks.</para>
464 <para>0: success</para>
468 <primary>LNet</primary>
469 <secondary>lustre_lnet_config_buf</secondary>
470 </indexterm>Adjusting Router Buffer Pools</title>
473 * lustre_lnet_config_buf
474 * Send down an IOCTL to configure buffer sizes. A value of 0 means
475 * default that particular buffer to default size. A value of -1 means
476 * leave the value of the buffer unchanged.
478 * tiny - tiny buffers
479 * small - small buffers
480 * large - large buffers.
481 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
483 extern int lustre_lnet_config_buf(int tiny,
487 cYAML **err_rc);</screen>
488 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
489 <para>IOC_LIBCFS_ADD_BUF</para>
490 <para><emphasis role="bold">Description:</emphasis></para>
491 <para>This API is used to configure the tiny, small and large
492 router buffers dynamically. These buffers are used to buffer
493 messages which are being routed to other nodes. The minimum value
494 of these buffers per CPT are:</para>
495 <screen>#define LNET_NRB_TINY_MIN 512
496 #define LNET_NRB_SMALL_MIN 4096
497 #define LNET_NRB_LARGE_MIN 256</screen>
498 <para>The default values of these buffers are:</para>
499 <screen>#define LNET_NRB_TINY (LNET_NRB_TINY_MIN * 4)
500 #define LNET_NRB_SMALL (LNET_NRB_SMALL_MIN * 4)
501 #define LNET_NRB_LARGE (LNET_NRB_LARGE_MIN * 4)</screen>
502 <para>These default value is divided evenly across all CPTs. However, each CPT can only go
503 as low as the minimum.</para>
504 <para>Multiple calls to this API with the same values has no effect</para>
505 <para><emphasis role="bold">Return Value</emphasis></para>
506 <para>-ENOMEM: if no memory to allocate buffer pools.</para>
507 <para>0: success</para>
511 <primary>LNet</primary>
512 <secondary>lustre_lnet_show_buf</secondary>
513 </indexterm>Showing Routing information</title>
516 * lustre_lnet_show_routing
517 * Send down an IOCTL to dump buffers and routing status
518 * This function is used to dump buffers for all CPU partitions.
520 * show_rc - [OUT] The show output in YAML. Must be freed by caller.
521 * err_rc - [OUT] struct cYAML tree describing the error. Freed by caller
523 extern int lustre_lnet_show_routing(int seq_no, struct cYAML **show_rc,
524 struct cYAML **err_rc);
526 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
527 <para>IOC_LIBCFS_GET_BUF</para>
528 <para><emphasis role="bold">Description:</emphasis></para>
529 <para>This API returns a cYAML block describing the values of each of the following per
534 <para>The number of pages per buffer. This is a constant.</para>
537 <para>The number of allocated buffers. This is a constant.</para>
540 <para>The number of buffer credits . This is a real-time value of the number of buffer
541 credits currently available. If this value is negative, that indicates the number of
542 queued messages.</para>
545 <para>The lowest number of credits ever reached in the system. This is historical
550 <para>The show block also returns the status of routing, whether enabled, or
552 <para>An exmaple YAML block</para>
571 <para><emphasis role="bold">Return Value</emphasis></para>
572 <para>-ENOMEM: if no memory to allocate the show or error block.</para>
573 <para>0: success</para>
577 <primary>LNet</primary>
578 <secondary>lustre_lnet_show stats</secondary>
579 </indexterm>Showing LNet Traffic Statistics</title>
582 * lustre_lnet_show_stats
583 * Shows internal LNet statistics. This is useful to display the
584 * current LNet activity, such as number of messages route, etc
586 * seq_no - sequence number of the command
587 * show_rc - YAML structure of the resultant show
588 * err_rc - YAML strucutre of the resultant return code.
590 extern int lustre_lnet_show_stats(int seq_no, cYAML **show_rc,
591 cYAML **err_rc);</screen>
592 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
593 <para>IOC_LIBCFS_GET_LNET_STATS</para>
594 <para><emphasis role="bold">Description:</emphasis></para>
595 <para>This API returns a cYAML block describing the LNet traffic
596 statistics. Statistics are continuously incremented by LNet while
597 it's alive. This API retuns the statistics at the time of the API
598 call. The statistics include the following</para>
602 <para>Number of messages allocated</para>
605 <para>Maximum number of messages in the system</para>
608 <para>Errors allocating or sending messages</para>
611 <para>Cumulative number of messages sent</para>
614 <para>Cumulative number of messages received</para>
617 <para>Cumulative number of messages routed</para>
620 <para>Cumulative number of messages dropped</para>
623 <para>Cumulative number of bytes sent</para>
626 <para>Cumulative number of bytes received</para>
629 <para>Cumulative number of bytes routed</para>
632 <para>Cumulative number of bytes dropped</para>
636 <para>An exmaple YAML block</para>
648 drop_length: 0</screen>
649 <para><emphasis role="bold">Return Value</emphasis></para>
650 <para>-ENOMEM: if no memory to allocate the show or error block.</para>
651 <para>0: success</para>
655 <primary>LNet</primary>
656 <secondary>lustre_yaml</secondary>
657 </indexterm>Adding/Deleting/Showing Parameters through a YAML Block</title>
661 * Parses the provided YAML file and then calls the specific APIs
662 * to configure the entities identified in the file
665 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
667 extern int lustre_yaml_config(char *f, cYAML **err_rc);
671 * Parses the provided YAML file and then calls the specific APIs
672 * to delete the entities identified in the file
675 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
677 extern int lustre_yaml_del(char *f, cYAML **err_rc);
681 * Parses the provided YAML file and then calls the specific APIs
682 * to show the entities identified in the file
685 * show_rc - [OUT] The show output in YAML. Must be freed by caller.
686 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
688 extern int lustre_yaml_show(char *f,
690 cYAML **err_rc);</screen>
691 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
692 <para>Depends on the entity being configured</para>
693 <para><emphasis role="bold">Description:</emphasis></para>
694 <para>These APIs add/remove/show the parameters specified in the
695 YAML file respectively. The entities don't have to be uniform.
696 Multiple different entities can be added/removed/showed in one
698 <para>An example YAML block</para>
701 - nid: 192.168.206.132@tcp
708 peer_buffer_credits: 0
713 gateway: 192.168.29.1@tcp
718 gateway: 192.168.28.1@tcp
727 <para><emphasis role="bold">Return Value</emphasis></para>
728 <para>Return value will correspond to the return value of the API
729 that will be called to operate on the configuration item, as
730 described in previous sections</para>
734 <primary>DLC</primary>
735 <secondary>Code Example</secondary>
736 </indexterm>Adding a route code example</title>
739 int main(int argc, char **argv)
741 char *network = NULL, *gateway = NULL;
742 long int hop = -1, prio = -1;
743 struct cYAML *err_rc = NULL;
747 const char *const short_options = "n:g:c:p:h";
748 const struct option long_options[] = {
749 { "net", 1, NULL, 'n' },
750 { "gateway", 1, NULL, 'g' },
751 { "hop-count", 1, NULL, 'c' },
752 { "priority", 1, NULL, 'p' },
753 { "help", 0, NULL, 'h' },
754 { NULL, 0, NULL, 0 },
757 while ((opt = getopt_long(argc, argv, short_options,
758 long_options, NULL)) != -1) {
767 rc = parse_long(optarg, &hop);
775 rc = parse_long(optarg, &prio);
783 print_help(route_cmds, "route", "add");
790 rc = lustre_lnet_config_route(network, gateway, hop, prio, -1, &err_rc);
792 if (rc != LUSTRE_CFG_RC_NO_ERR)
793 cYAML_print_tree2file(stderr, err_rc);
795 cYAML_free_tree(err_rc);
799 <para>For other code examples refer to
800 <screen>lnet/utils/lnetctl.c</screen></para>