1 ******************************************
2 * Overview of Dynamic LNet Configuration *
3 ******************************************
7 Amir Shehata <amir.shehata@intel.com>
16 4. LNet Backend Design
18 4.1.1 Enabling/Disabling Routing and Router Pools
19 4.1.2 Configuring Router Pools
21 4.2.1 Dynamically Adding an NI
22 4.2.2 Dynamically Deleting an NI
23 4.2.3 Internal Functions
32 Dynamic LNet Configuration (DLC) strives to accomplish four objectives:
34 . Allow changes to some LNet parameters to be dynamic so they can be altered
35 without having to stop/start LNet.
36 . Remove the limits on how many NI's or route's which can be configured.
37 . Work towards replacing the LNet module parameters with user space scripts
38 (i.e. /etc/rc.d and /etc/sysconfig/network-scripts) which operate much more
39 like conventional network config systems on Linux.
40 . Allow sysadmins to use a command line utility for making dynamic changes to
41 the LNet configuration.
46 DLC introduces a new shared library. The DLC C API Library is provided as part
47 of the standard lustre installation. This library must be released under LGPL
48 license to facilitate third-party development. This API will serve the
51 . Provide a funnel for all LNet configurations. This way there isn't multiple
52 ways to configure the same thing
53 . Provide a C API which configures specific parameters. This C API will
54 translate into IOCTL calls to the kernel module.
55 . Provide a YAML interface, which ensures that YAML parsing is done in one
57 . Provide more detailed return values from the kernel presented in YAML
59 . Provide a set of API calls to parse out the YAML error returns if need be.
64 DLC introduces a userspace library which exposes a set of APIs to:
66 . configure/unconfigure LNet
67 . add/delete/show routes
68 . add/delete/show networks
69 . set router buffers sizes
70 . enable/disable routing
71 . import YAML configuration
72 . export configuration in YAML format
73 . show peer credit information
75 DLC API uses IOCTLs to communicate with the kernel. This has the following
78 . No need to do any string parsing in the kernel
79 . Bypass the 4K limit, since each command will be sent to the kernel
80 separately. For example, if there are 100 routes to configure then this
81 translates to 100 ioctls to the kernel.
82 . Provide a more intuitive user facing interface. The user need not know how
83 to talk to the kernel via ioctl. This will be achieved in two ways:
84 a. The configuration can be presented in YAML syntax, as shown below
85 b. A more traditional C API will be exported, Each translates to exactly
86 one ioctl to the kernel module.
92 The DLC API provides functions to configure specific parameters together with
93 higher level functions which accept a YAML formatted configuration and
94 configure all parameters described therein.
96 To aid in parsing and building YAML stings a third party library, libyaml, is
97 used. lnet/utils/cyaml/cyaml.c is written to interface with the libyaml
98 library. When parsing a yaml string or file, it builds an internal tree
99 representation of the YAML string. lnet/utils/cyaml/cyaml.c then provides
100 functions to extract data from the tree. It also provides functions to build
101 a YAML tree, which can then be printed textually.
103 The YAML API is used by the DLC API to parse and extract YAML configuration,
104 and to build a YAML representation of the success or error status per DLC API.
110 Refer to: lnet/utils/lnetconfig/liblnetconfig.h
112 ==========================
113 = 4. LNet Backend Design =
114 ==========================
116 4.1 lnet/lnet/router.c
117 =======================
119 In router.c the following three functions are used to add, delete and get a
122 . lnet_add_route() : adds a new route entry.
123 . lnet_del_route() : removes an existing routing entry.
124 . lnet_get_route() : retrieves information on an existing route.
126 These functions will be called from the ioctl handler.
128 The Router buffer pools will be controlled via the following APIs:
130 . lnet_rtrpools_enable() : enable routing and allocate router buffers
131 . lnet_rtrpools_disable() : disable routing and free router buffers
132 . lnet_rtrpools_adjust() : configure the number of router buffer pools
134 4.1.1 Enabling/Disabling Routing and Router Pools
135 =================================================
137 lnet_rtrpools_enable()
138 lnet_rtrpools_disable()
140 These functions will explicitly enable and disable rtrpools respectively. If the
141 rtrpools are being enabled then the default values of the tiny, small and
142 large buffers are used.
144 #define LNET_NRB_TINY (LNET_NRB_TINY_MIN * 4)
145 #define LNET_NRB_SMALL (LNET_NRB_SMALL_MIN * 4)
146 #define LNET_NRB_LARGE (LNET_NRB_LARGE_MIN * 4)
148 4.1.2 Configuring Router Pools
149 ==============================
151 lnet_rtrpools_adjust()
153 This API will set the tiny, small and large buffers. rtrpools must be enabled
154 prior to calling this API. If not enabled, this API is a no-op.
156 Below is the behavior when adjusting the router pools:
158 . A value of 0 indicates that the default values (listed above) are used.
159 . Each pool size will be run through the current sizing algorithm to ensure the
160 given values are not below the minimum standards. As such, a user could get
161 the defaults for all three pools by just passing in "0 0 0" for pool sizes.
162 . If a pool size is being made bigger, add the new buffers to the pool checking
163 if there are any queued messages waiting for a buffer on that pool. If so,
164 schedule those messages to be sent with the new buffer.
165 . If a pool size is being made smaller, change the maximum value for the number
166 of buffers on the pool only. When the system returns buffers to the pool, it
167 will see there are "excess" buffers on the pool and will discard the buffer
168 rather than return it to the pool.
170 4.2 lnet/lnet/api-ni.c
171 =======================
173 The file api-ni.c contains the interface routines for working with network
174 interfaces. Four new routines will be added to this file to support this
177 lnet_dyn_add_ni() : adds an NI dynamically
178 lnet_dyn_del_ni() : deletes an NI dynamically
179 lnet_startup_lndni() : starts a single LND.
180 lnet_shutdown_lndni() : shuts down a single LND.
182 4.2.1 Dynamically Adding an NI
183 ==============================
187 This function is called to startup a network passed in through userspace as a
188 string. This function verifies that only one network is passed in. If more
189 than one network is passed in, it fails with -EINVAL error number.
191 Peer timeout, peer credits, and credits are passed in as well, since DLC makes
192 these tunable parameters configurable per network interface.
194 This function does the following:
196 . Parses the network string passed in.
197 . Initializes a new ping info structure in preparation of adding the Network
200 . Updates the global ping info.
201 . If there is any failure in any of these steps, appropriate cleanup action is
204 4.2.2 Dynamically Deleting an NI
205 ================================
209 This function does the following:
211 . Allocates a new ping info for the number of networks without the one being
214 . Updates the global ping info appropriately.
216 4.2.3 Internal Functions
217 ========================
221 The code was restructured to breakup the function lnet_startup_lndnis() into
222 two functions: lnet_startup_lndni() and lnet_startup_lndnis(). This way the
223 new lnet_dyn_add_ni() can call lnet_startup_lndni() to startup one NI, and on
224 standard startup lnet_startup_lndnis() can start up all NIs configured in the
225 module parameters, by calling lnet_startup_lndni() repeatedly.
227 lnet_shutdown_lndni()
229 This function was introduced to shutdown only one NI. It coexists with the
230 original lnet_shutdown_lndnis() which shuts down all NIs, which is needed when
231 LNet is being unconfigured.
233 lnet_shutdown_lndni() is called from lnet_dyn_del_ni() and from
234 lnet_startup_lndni() in case there is a failure starting the NI.