1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="lnetconfigurationapi">
5 <title xml:id="lnetconfigurationapi.title">LNet Configuration C-API</title>
6 <para>This section describes the LNet Configuration C-API library. This
7 API allows the developer to programatically configure LNet. It provides
8 APIs to add, delete and show LNet configuration items listed below. The
9 API utilizes IOCTL to communicate with the kernel. Changes take effect
10 immediately and do not require restarting LNet. API calls are
15 <para>Configuring LNet</para>
18 <para>Enabling/Disabling routing</para>
21 <para>Adding/removing/showing Routes</para>
24 <para>Adding/removing/showing Networks</para>
27 <para>Configuring Router Buffer Pools</para>
33 <primary>LNet</primary>
34 <secondary>capi general information</secondary>
35 </indexterm>General API Information</title>
39 <primary>LNet</primary>
40 <secondary>capi return code</secondary>
41 </indexterm>API Return Code</title>
42 <screen>LUSTRE_CFG_RC_NO_ERR 0
43 LUSTRE_CFG_RC_BAD_PARAM -1
44 LUSTRE_CFG_RC_MISSING_PARAM -2
45 LUSTRE_CFG_RC_OUT_OF_RANGE_PARAM -3
46 LUSTRE_CFG_RC_OUT_OF_MEM -4
47 LUSTRE_CFG_RC_GENERIC_ERR -5</screen>
51 <primary>LNet</primary>
52 <secondary>capi input params</secondary>
53 </indexterm>API Common Input Parameters</title>
54 <para>All APIs take as input a sequence number. This is a number
55 that's assigned by the caller of the API, and is returned in the
56 YAML error return block. It is used to associate the request with
57 the response. It is especially useful when configuring via the
58 YAML interface, since typically the YAML interface is used to
59 configure multiple items. In the
60 return Error block, it is desired to know which items were
61 configured properly and which were not configured properly. The
62 sequence number achieves this purpose.</para>
66 <primary>LNet</primary>
67 <secondary>capi output params</secondary>
68 </indexterm>API Common Output Parameters</title>
72 <primary>LNet</primary>
73 <secondary>cyaml</secondary>
74 </indexterm>Internal YAML Representation (cYAML)</title>
75 <para>Once a YAML block is parsed it needs to be stored
76 structurally in order to facilitate passing it to different
77 functions, querying it and printing it. Also it is required to
78 be able to build this internal representation from data returned
79 from the kernel and return it to the caller, which can query and
80 print it. This structure
81 representation is used for the Error and Show API Out
82 parameters. For this YAML is internally represented via this
84 <screen>typedef enum {
85 EN_YAML_TYPE_FALSE = 0,
92 } cYAML_object_type_t;
94 typedef struct cYAML {
95 /* next/prev allow you to walk array/object chains. */
96 struct cYAML *cy_next, *cy_prev;
97 /* An array or object item will have a child pointer pointing
98 to a chain of the items in the array/object. */
99 struct cYAML *cy_child;
100 /* The type of the item, as above. */
101 cYAML_object_type_t cy_type;
102 /* The item's string, if type==EN_YAML_TYPE_STRING */
103 char *cy_valuestring;
104 /* The item's number, if type==EN_YAML_TYPE_NUMBER */
106 /* The item's number, if type==EN_YAML_TYPE_NUMBER */
107 double cy_valuedouble;
108 /* The item's name string, if this item is the child of,
109 or is in the list of subitems of an object. */
111 /* user data which might need to be tracked per object */
117 <primary>LNet</primary>
118 <secondary>error block</secondary>
119 </indexterm>Error Block</title>
120 <para>All APIs return a cYAML error block. This error block has
121 the following format, when it's printed out. All configuration
122 errors shall be represented in a YAML sequence</para>
125 errno: <error number>
126 seqno: <sequence number>
127 descr: <error description>
134 descr: Missing mandatory parameter(s) - network</screen>
138 <primary>LNet</primary>
139 <secondary>show block</secondary>
140 </indexterm>Show Block</title>
141 <para>All Show APIs return a cYAML show block. This show block
142 represents the information requested in YAML format. Each
143 configuration item has its own YAML syntax. The YAML syntax of
144 all supported configuration items is described later in this
145 document. Below is an example of a show block:</para>
147 - nid: 192.168.206.130@tcp4
154 peer_buffer_credits: 30
161 <primary>LNet</primary>
162 <secondary>show block</secondary>
163 </indexterm>The LNet Configuration C-API</title>
167 <primary>LNet</primary>
168 <secondary>lustre_lnet_config_ni_system</secondary>
169 </indexterm>Configuring LNet</title>
173 * lustre_lnet_config_ni_system
174 * Initialize/Uninitialize the LNet NI system.
176 * up - whether to init or uninit the system
177 * load_ni_from_mod - load NI from mod params.
178 * seq_no - sequence number of the request
179 * err_rc - [OUT] struct cYAML tree describing the error. Freed by
182 int lustre_lnet_config_ni_system(bool up, bool load_ni_from_mod,
183 int seq_no, struct cYAML **err_rc);</screen>
184 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
185 <para>IOC_LIBCFS_CONFIGURE or IOC_LIBCFS_UNCONFIGURE</para>
186 <para><emphasis role="bold">Description:</emphasis></para>
187 <para><emphasis role="bold">Configuring LNet</emphasis>
189 <para>Initialize LNet internals and load any networks specified in the module
190 parameter if <literal>load_ni_from_mod</literal> is set. Otherwise do not
191 load any network interfaces.</para>
192 <para><emphasis role="bold">Unconfiguring LNet</emphasis></para>
193 <para>Bring down LNet and clean up network itnerfaces, routes and all LNet
195 <para><emphasis role="bold">Return Value</emphasis></para>
196 <para>0: if success</para>
197 <para>-errno: if failure</para>
201 <primary>LNet</primary>
202 <secondary>lustre_lnet_enable_routing</secondary>
203 </indexterm>Enabling and Disabling Routing</title>
206 * lustre_lnet_enable_routing
207 * Send down an IOCTL to enable or disable routing
209 * enable - 1 to enable routing, 0 to disable routing
210 * seq_no - sequence number of the request
211 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
213 extern int lustre_lnet_enable_routing(int enable,
215 cYAML **err_rc);</screen>
216 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
217 <para>IOC_LIBCFS_ENABLE_RTR </para>
218 <para><emphasis role="bold">Description:</emphasis></para>
219 <para><emphasis role="bold">Enabling Routing</emphasis>
221 <para>The router buffer pools are allocated using the default values. Internally the node
222 is then flagged as a Router node. The node can be used as a router from this
224 <para><emphasis role="bold">Disabling Routing</emphasis></para>
225 <para>The unused router buffer pools are freed. Buffers currently
226 in use are not freed until they are returned to the unused list.
227 Internally the node routing flag is turned off. Any subsequent
228 messages not destined to this node are dropped. </para>
229 <para><emphasis role="bold">Enabling Routing on an already enabled
230 node, or vice versa</emphasis></para>
231 <para>In both these cases the LNet Kernel module ignores this request. </para>
232 <para><emphasis role="bold">Return Value</emphasis></para>
233 <para>-ENOMEM: if there is no memory to allocate buffer pools</para>
234 <para>0: if success</para>
238 <primary>LNet</primary>
239 <secondary>lustre_lnet_config_route</secondary>
240 </indexterm>Adding Routes</title>
243 * lustre_lnet_config_route
244 * Send down an IOCTL to the kernel to configure the route
248 * hops - number of hops passed down by the user
249 * prio - priority of the route
250 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
252 extern int lustre_lnet_config_route(char *nw, char *gw,
255 cYAML **err_rc);</screen>
256 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
257 <para>IOC_LIBCFS_ADD_ROUTE</para>
258 <para><emphasis role="bold">Description:</emphasis></para>
259 <para>The LNet Kernel module adds this route to the list of
260 existing routes, if one doesn't already exist. If hop parameter is
261 not specified (IE: -1) then the hop count is set to 1. If the
262 priority parameter is not specified (IE: -1) then the priority is
263 set to 0. All routes with the same hop and priority are used in
264 round robin. Routes with lower number of hops and/or higher
265 priority are preferred. 0 is the highest priority.</para>
266 <para>If a route already exists the request to add the same route is ignored.</para>
267 <para><emphasis role="bold">Return Value</emphasis></para>
268 <para>-EINVAL: if the network of the route is local</para>
269 <para>-ENOMEM: if there is no memory</para>
270 <para>-EHOSTUNREACH: if the host is not on a local network</para>
271 <para>0: if success</para>
275 <primary>LNet</primary>
276 <secondary>lustre_lnet_del_route</secondary>
277 </indexterm>Deleting Routes</title>
280 * lustre_lnet_del_route
281 * Send down an IOCTL to the kernel to delete a route
286 extern int lustre_lnet_del_route(char *nw, char *gw,
288 cYAML **err_rc);</screen>
289 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
290 <para>IOC_LIBCFS_DEL_ROUTE</para>
291 <para><emphasis role="bold">Description:</emphasis></para>
292 <para>LNet will remove the route which matches the network and gateway passed in. If
293 no route matches, then the operation fails with an appropriate error number.</para>
294 <para><emphasis role="bold">Return Value</emphasis></para>
295 <para>-ENOENT: if the entry being deleted doesn't exist</para>
296 <para>0: if success</para>
300 <primary>LNet</primary>
301 <secondary>lustre_lnet_show_route</secondary>
302 </indexterm>Showing Routes</title>
305 * lustre_lnet_show_route
306 * Send down an IOCTL to the kernel to show routes
307 * This function will get one route at a time and filter according to
308 * provided parameters. If no filter is provided then it will dump all
309 * routes that are in the system.
311 * nw - network. Optional. Used to filter output
312 * gw - gateway. Optional. Used to filter ouptut
313 * hops - number of hops passed down by the user
314 * Optional. Used to filter output.
315 * prio - priority of the route. Optional. Used to filter output.
316 * detail - flag to indicate whether detail output is required
317 * show_rc - [OUT] The show output in YAML. Must be freed by caller.
318 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
320 extern int lustre_lnet_show_route(char *nw, char *gw,
321 int hops, int prio, int detail,
324 cYAML **err_rc);</screen>
325 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
326 <para>IOC_LIBCFS_GET_ROUTE</para>
327 <para><emphasis role="bold">Description:</emphasis></para>
328 <para>The routes are fetched from the kernel one by one and packed
329 in a cYAML block, after filtering according to the parameters
330 passed in. The cYAML block is then returned to the caller of the
332 <para>An example with the detail parameter set to 1</para>
335 gateway: 192.168.205.130@tcp
339 <para>An Example with the detail parameter set to 0</para>
342 gateway: 192.168.205.130@tcp</screen>
343 <para><emphasis role="bold">Return Value</emphasis></para>
344 <para>-ENOMEM: If no memory</para>
345 <para>0: if success</para>
349 <primary>LNet</primary>
350 <secondary>lustre_lnet_config_net</secondary>
351 </indexterm>Adding a Network Interface</title>
354 * lustre_lnet_config_net
355 * Send down an IOCTL to configure a network.
357 * net - the network name
358 * intf - the interface of the network of the form net_name(intf)
359 * peer_to - peer timeout
360 * peer_cr - peer credit
361 * peer_buf_cr - peer buffer credits
362 * - the above are LND tunable parameters and are optional
363 * credits - network interface credits
365 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
367 extern int lustre_lnet_config_net(char *net,
375 cYAML **err_rc);</screen>
376 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
377 <para>IOC_LIBCFS_ADD_NET</para>
378 <para><emphasis role="bold">Description:</emphasis></para>
379 <para>A new network is added and initialized. This has the same
380 effect as configuring a network from the module parameters. The
381 API allows the specification of network parameters such as the
382 peer timeout, peer credits, peer buffer credits and credits. The
383 CPU affinity of the network interface being added can also be
384 specified. These parameters become
385 network specific under Dynamic LNet Configuration (DLC), as
386 opposed to being per LND as it was previously.</para>
387 <para>If an already existing network is added the request is ignored.</para>
388 <para><emphasis role="bold">Return Value</emphasis></para>
389 <para>-EINVAL: if the network passed in is not recognized.</para>
390 <para>-ENOMEM: if no memory</para>
391 <para>0: success</para>
395 <primary>LNet</primary>
396 <secondary>lustre_lnet_del_net</secondary>
397 </indexterm>Deleting a Network Interface</title>
400 * lustre_lnet_del_net
401 * Send down an IOCTL to delete a network.
403 * nw - network to delete.
404 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
406 extern int lustre_lnet_del_net(char *nw,
408 cYAML **err_rc);</screen>
409 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
410 <para>IOC_LIBCFS_DEL_NET</para>
411 <para><emphasis role="bold">Description:</emphasis></para>
412 <para>The network interface specified is deleted. All resources
413 associated with this network interface are freed. All routes going
414 over that Network Interface are cleaned up.</para>
415 <para>If a non existent network is deleted then the call return -EINVAL.</para>
416 <para><emphasis role="bold">Return Value</emphasis></para>
417 <para>-EINVAL: if the request references a non-existent network.</para>
418 <para>0: success</para>
422 <primary>LNet</primary>
423 <secondary>lustre_lnet_show_net</secondary>
424 </indexterm>Showing Network Interfaces</title>
427 * lustre_lnet_show_net
428 * Send down an IOCTL to show networks.
429 * This function will use the nw paramter to filter the output. If it's
430 * not provided then all networks are listed.
432 * nw - network to show. Optional. Used to filter output.
433 * detail - flag to indicate if we require detail output.
434 * show_rc - [OUT] The show output in YAML. Must be freed by caller.
435 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
437 extern int lustre_lnet_show_net(char *nw, int detail,
440 cYAML **err_rc);</screen>
441 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
442 <para>IOC_LIBCFS_GET_NET</para>
443 <para><emphasis role="bold">Description:</emphasis></para>
444 <para>The network interfaces are queried one at a time from the
445 kernel and packed in a cYAML block, after filtering on the network
446 (EX: tcp). If the detail field is set to 1, then the tunable
447 section of the show block is included in the return.</para>
448 <para>An example of the detailed output</para>
450 nid: 192.168.206.130@tcp4
457 peer_buffer_credits: 30
459 <para>An example of none detailed output</para>
461 nid: 192.168.206.130@tcp4
464 intf-0: eth0</screen>
465 <para><emphasis role="bold">Return Value</emphasis></para>
466 <para>-ENOMEM: if no memory to allocate the error or show blocks.</para>
467 <para>0: success</para>
471 <primary>LNet</primary>
472 <secondary>lustre_lnet_config_buf</secondary>
473 </indexterm>Adjusting Router Buffer Pools</title>
476 * lustre_lnet_config_buf
477 * Send down an IOCTL to configure buffer sizes. A value of 0 means
478 * default that particular buffer to default size. A value of -1 means
479 * leave the value of the buffer unchanged.
481 * tiny - tiny buffers
482 * small - small buffers
483 * large - large buffers.
484 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
486 extern int lustre_lnet_config_buf(int tiny,
490 cYAML **err_rc);</screen>
491 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
492 <para>IOC_LIBCFS_ADD_BUF</para>
493 <para><emphasis role="bold">Description:</emphasis></para>
494 <para>This API is used to configure the tiny, small and large
495 router buffers dynamically. These buffers are used to buffer
496 messages which are being routed to other nodes. The minimum value
497 of these buffers per CPT are:</para>
498 <screen>#define LNET_NRB_TINY_MIN 512
499 #define LNET_NRB_SMALL_MIN 4096
500 #define LNET_NRB_LARGE_MIN 256</screen>
501 <para>The default values of these buffers are:</para>
502 <screen>#define LNET_NRB_TINY (LNET_NRB_TINY_MIN * 4)
503 #define LNET_NRB_SMALL (LNET_NRB_SMALL_MIN * 4)
504 #define LNET_NRB_LARGE (LNET_NRB_LARGE_MIN * 4)</screen>
505 <para>These default value is divided evenly across all CPTs. However, each CPT can only go
506 as low as the minimum.</para>
507 <para>Multiple calls to this API with the same values has no effect</para>
508 <para><emphasis role="bold">Return Value</emphasis></para>
509 <para>-ENOMEM: if no memory to allocate buffer pools.</para>
510 <para>0: success</para>
514 <primary>LNet</primary>
515 <secondary>lustre_lnet_show_buf</secondary>
516 </indexterm>Showing Routing information</title>
519 * lustre_lnet_show_routing
520 * Send down an IOCTL to dump buffers and routing status
521 * This function is used to dump buffers for all CPU partitions.
523 * show_rc - [OUT] The show output in YAML. Must be freed by caller.
524 * err_rc - [OUT] struct cYAML tree describing the error. Freed by caller
526 extern int lustre_lnet_show_routing(int seq_no, struct cYAML **show_rc,
527 struct cYAML **err_rc);
529 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
530 <para>IOC_LIBCFS_GET_BUF</para>
531 <para><emphasis role="bold">Description:</emphasis></para>
532 <para>This API returns a cYAML block describing the values of each of the following per
537 <para>The number of pages per buffer. This is a constant.</para>
540 <para>The number of allocated buffers. This is a constant.</para>
543 <para>The number of buffer credits . This is a real-time value of the number of buffer
544 credits currently available. If this value is negative, that indicates the number of
545 queued messages.</para>
548 <para>The lowest number of credits ever reached in the system. This is historical
553 <para>The show block also returns the status of routing, whether enabled, or
555 <para>An exmaple YAML block</para>
574 <para><emphasis role="bold">Return Value</emphasis></para>
575 <para>-ENOMEM: if no memory to allocate the show or error block.</para>
576 <para>0: success</para>
580 <primary>LNet</primary>
581 <secondary>lustre_lnet_show stats</secondary>
582 </indexterm>Showing LNet Traffic Statistics</title>
585 * lustre_lnet_show_stats
586 * Shows internal LNet statistics. This is useful to display the
587 * current LNet activity, such as number of messages route, etc
589 * seq_no - sequence number of the command
590 * show_rc - YAML structure of the resultant show
591 * err_rc - YAML strucutre of the resultant return code.
593 extern int lustre_lnet_show_stats(int seq_no, cYAML **show_rc,
594 cYAML **err_rc);</screen>
595 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
596 <para>IOC_LIBCFS_GET_LNET_STATS</para>
597 <para><emphasis role="bold">Description:</emphasis></para>
598 <para>This API returns a cYAML block describing the LNet traffic
599 statistics. Statistics are continuously incremented by LNet while
600 it's alive. This API retuns the statistics at the time of the API
601 call. The statistics include the following</para>
605 <para>Number of messages allocated</para>
608 <para>Maximum number of messages in the system</para>
611 <para>Errors allocating or sending messages</para>
614 <para>Cumulative number of messages sent</para>
617 <para>Cumulative number of messages received</para>
620 <para>Cumulative number of messages routed</para>
623 <para>Cumulative number of messages dropped</para>
626 <para>Cumulative number of bytes sent</para>
629 <para>Cumulative number of bytes received</para>
632 <para>Cumulative number of bytes routed</para>
635 <para>Cumulative number of bytes dropped</para>
639 <para>An exmaple YAML block</para>
651 drop_length: 0</screen>
652 <para><emphasis role="bold">Return Value</emphasis></para>
653 <para>-ENOMEM: if no memory to allocate the show or error block.</para>
654 <para>0: success</para>
658 <primary>LNet</primary>
659 <secondary>lustre_yaml</secondary>
660 </indexterm>Adding/Deleting/Showing Parameters through a YAML Block</title>
664 * Parses the provided YAML file and then calls the specific APIs
665 * to configure the entities identified in the file
668 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
670 extern int lustre_yaml_config(char *f, cYAML **err_rc);
674 * Parses the provided YAML file and then calls the specific APIs
675 * to delete the entities identified in the file
678 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
680 extern int lustre_yaml_del(char *f, cYAML **err_rc);
684 * Parses the provided YAML file and then calls the specific APIs
685 * to show the entities identified in the file
688 * show_rc - [OUT] The show output in YAML. Must be freed by caller.
689 * err_rc - [OUT] cYAML tree describing the error. Freed by caller
691 extern int lustre_yaml_show(char *f,
693 cYAML **err_rc);</screen>
694 <para><emphasis role="bold">IOCTL to Kernel:</emphasis></para>
695 <para>Depends on the entity being configured</para>
696 <para><emphasis role="bold">Description:</emphasis></para>
697 <para>These APIs add/remove/show the parameters specified in the
698 YAML file respectively. The entities don't have to be uniform.
699 Multiple different entities can be added/removed/showed in one
701 <para>An example YAML block</para>
704 - nid: 192.168.206.132@tcp
711 peer_buffer_credits: 0
716 gateway: 192.168.29.1@tcp
721 gateway: 192.168.28.1@tcp
730 <para><emphasis role="bold">Return Value</emphasis></para>
731 <para>Return value will correspond to the return value of the API
732 that will be called to operate on the configuration item, as
733 described in previous sections</para>
737 <primary>DLC</primary>
738 <secondary>Code Example</secondary>
739 </indexterm>Adding a route code example</title>
742 int main(int argc, char **argv)
744 char *network = NULL, *gateway = NULL;
745 long int hop = -1, prio = -1;
746 struct cYAML *err_rc = NULL;
750 const char *const short_options = "n:g:c:p:h";
751 const struct option long_options[] = {
752 { "net", 1, NULL, 'n' },
753 { "gateway", 1, NULL, 'g' },
754 { "hop-count", 1, NULL, 'c' },
755 { "priority", 1, NULL, 'p' },
756 { "help", 0, NULL, 'h' },
757 { NULL, 0, NULL, 0 },
760 while ((opt = getopt_long(argc, argv, short_options,
761 long_options, NULL)) != -1) {
770 rc = parse_long(optarg, &hop);
778 rc = parse_long(optarg, &prio);
786 print_help(route_cmds, "route", "add");
793 rc = lustre_lnet_config_route(network, gateway, hop, prio, -1, &err_rc);
795 if (rc != LUSTRE_CFG_RC_NO_ERR)
796 cYAML_print_tree2file(stderr, err_rc);
798 cYAML_free_tree(err_rc);
802 <para>For other code examples refer to
803 <screen>lnet/utils/lnetctl.c</screen></para>
807 <!--vim:expandtab:shiftwidth=2:tabstop=8:-->