From: Amir Shehata Date: Thu, 15 Jan 2015 18:30:24 +0000 (-0800) Subject: LU-6129 lnet: DLC design doc X-Git-Tag: 2.7.51~30 X-Git-Url: https://git.whamcloud.com/?p=fs%2Flustre-release.git;a=commitdiff_plain;h=ddaa254a64e2dfb875ecb3715a8dbf2053e13bd9 LU-6129 lnet: DLC design doc Add a DLC design document to lustre/doc/ Signed-off-by: Amir Shehata Change-Id: I3ec283f960aba7b56afadcc1d2a7770604efb023 Reviewed-on: http://review.whamcloud.com/13419 Tested-by: Jenkins Reviewed-by: John L. Hammond Reviewed-by: Andreas Dilger --- diff --git a/lustre/doc/dlc.txt b/lustre/doc/dlc.txt new file mode 100644 index 0000000..3ca0a14 --- /dev/null +++ b/lustre/doc/dlc.txt @@ -0,0 +1,235 @@ + ****************************************** + * Overview of Dynamic LNet Configuration * + ****************************************** + +Original Authors: +================= +Amir Shehata + +Contents +1. Introduction + 1.1 Objectives + 1.3 Motivation + 1.2 Overview +2. YAML Interface +3. DLC API +4. LNet Backend Design + 4.1 router.c + 4.1.1 Enabling/Disabling Routing and Router Pools + 4.1.2 Configuring Router Pools + 4.2 api-ni.c + 4.2.1 Dynamically Adding an NI + 4.2.2 Dynamically Deleting an NI + 4.2.3 Internal Functions + +=================== += 1. Introduction = +=================== + +1.1 Objectives +============== + +Dynamic LNet Configuration (DLC) strives to accomplish four objectives: + +. Allow changes to some LNet parameters to be dynamic so they can be altered + without having to stop/start LNet. +. Remove the limits on how many NI's or route's which can be configured. +. Work towards replacing the LNet module parameters with user space scripts + (i.e. /etc/rc.d and /etc/sysconfig/network-scripts) which operate much more + like conventional network config systems on Linux. +. Allow sysadmins to use a command line utility for making dynamic changes to + the LNet configuration. + +1.2 Motivation +============== + +DLC introduces a new shared library. The DLC C API Library is provided as part +of the standard lustre installation. This library must be released under LGPL +license to facilitate third-party development. This API will serve the +following purposes: + +. Provide a funnel for all LNet configurations. This way there isn't multiple + ways to configure the same thing +. Provide a C API which configures specific parameters. This C API will + translate into IOCTL calls to the kernel module. +. Provide a YAML interface, which ensures that YAML parsing is done in one + location. +. Provide more detailed return values from the kernel presented in YAML + format. +. Provide a set of API calls to parse out the YAML error returns if need be. + +1.3 Overview +============ + +DLC introduces a userspace library which exposes a set of APIs to: + +. configure/unconfigure LNet +. add/delete/show routes +. add/delete/show networks +. set router buffers sizes +. enable/disable routing +. import YAML configuration +. export configuration in YAML format +. show peer credit information + +DLC API uses IOCTLs to communicate with the kernel. This has the following +advantage: + +. No need to do any string parsing in the kernel +. Bypass the 4K limit, since each command will be sent to the kernel + separately. For example, if there are 100 routes to configure then this + translates to 100 ioctls to the kernel. +. Provide a more intuitive user facing interface. The user need not know how + to talk to the kernel via ioctl. This will be achieved in two ways: + a. The configuration can be presented in YAML syntax, as shown below + b. A more traditional C API will be exported, Each translates to exactly + one ioctl to the kernel module. + +===================== += 2. YAML Interface = +===================== + +The DLC API provides functions to configure specific parameters together with +higher level functions which accept a YAML formatted configuration and +configure all parameters described therein. + +To aid in parsing and building YAML stings a third party library, libyaml, is +used. lnet/utils/cyaml/cyaml.c is written to interface with the libyaml +library. When parsing a yaml string or file, it builds an internal tree +representation of the YAML string. lnet/utils/cyaml/cyaml.c then provides +functions to extract data from the tree. It also provides functions to build +a YAML tree, which can then be printed textually. + +The YAML API is used by the DLC API to parse and extract YAML configuration, +and to build a YAML representation of the success or error status per DLC API. + +============== += 3. DLC API = +============== + +Refer to: lnet/utils/lnetconfig/liblnetconfig.h + +========================== += 4. LNet Backend Design = +========================== + +4.1 lnet/lnet/router.c +======================= + +In router.c the following three functions are used to add, delete and get a +route. + +. lnet_add_route() : adds a new route entry. +. lnet_del_route() : removes an existing routing entry. +. lnet_get_route() : retrieves information on an existing route. + +These functions will be called from the ioctl handler. + +The Router buffer pools will be controlled via the following APIs: + +. lnet_rtrpools_enable() : enable routing and allocate router buffers +. lnet_rtrpools_disable() : disable routing and free router buffers +. lnet_rtrpools_adjust() : configure the number of router buffer pools + +4.1.1 Enabling/Disabling Routing and Router Pools +================================================= + +lnet_rtrpools_enable() +lnet_rtrpools_disable() + +These functions will explicitly enable and disable rtrpools respectively. If the +rtrpools are being enabled then the default values of the tiny, small and +large buffers are used. + + #define LNET_NRB_TINY (LNET_NRB_TINY_MIN * 4) + #define LNET_NRB_SMALL (LNET_NRB_SMALL_MIN * 4) + #define LNET_NRB_LARGE (LNET_NRB_LARGE_MIN * 4) + +4.1.2 Configuring Router Pools +============================== + +lnet_rtrpools_adjust() + +This API will set the tiny, small and large buffers. rtrpools must be enabled +prior to calling this API. If not enabled, this API is a no-op. + +Below is the behavior when adjusting the router pools: + +. A value of 0 indicates that the default values (listed above) are used. +. Each pool size will be run through the current sizing algorithm to ensure the + given values are not below the minimum standards. As such, a user could get + the defaults for all three pools by just passing in "0 0 0" for pool sizes. +. If a pool size is being made bigger, add the new buffers to the pool checking + if there are any queued messages waiting for a buffer on that pool. If so, + schedule those messages to be sent with the new buffer. +. If a pool size is being made smaller, change the maximum value for the number + of buffers on the pool only. When the system returns buffers to the pool, it + will see there are "excess" buffers on the pool and will discard the buffer + rather than return it to the pool. + +4.2 lnet/lnet/api-ni.c +======================= + +The file api-ni.c contains the interface routines for working with network +interfaces. Four new routines will be added to this file to support this +feature. + +lnet_dyn_add_ni() : adds an NI dynamically +lnet_dyn_del_ni() : deletes an NI dynamically +lnet_startup_lndni() : starts a single LND. +lnet_shutdown_lndni() : shuts down a single LND. + +4.2.1 Dynamically Adding an NI +============================== + +lnet_dyn_add_ni() + +This function is called to startup a network passed in through userspace as a +string. This function verifies that only one network is passed in. If more +than one network is passed in, it fails with -EINVAL error number. + +Peer timeout, peer credits, and credits are passed in as well, since DLC makes +these tunable parameters configurable per network interface. + +This function does the following: + +. Parses the network string passed in. +. Initializes a new ping info structure in preparation of adding the Network + Interface. +. Starts up the NI +. Updates the global ping info. +. If there is any failure in any of these steps, appropriate cleanup action is + taken. + +4.2.2 Dynamically Deleting an NI +================================ + +lnet_dyn_del_ni() + +This function does the following: + +. Allocates a new ping info for the number of networks without the one being + removed. +. Shuts down the NI +. Updates the global ping info appropriately. + +4.2.3 Internal Functions +======================== + +lnet_startup_lndni() + +The code was restructured to breakup the function lnet_startup_lndnis() into +two functions: lnet_startup_lndni() and lnet_startup_lndnis(). This way the +new lnet_dyn_add_ni() can call lnet_startup_lndni() to startup one NI, and on +standard startup lnet_startup_lndnis() can start up all NIs configured in the +module parameters, by calling lnet_startup_lndni() repeatedly. + +lnet_shutdown_lndni() + +This function was introduced to shutdown only one NI. It coexists with the +original lnet_shutdown_lndnis() which shuts down all NIs, which is needed when +LNet is being unconfigured. + +lnet_shutdown_lndni() is called from lnet_dyn_del_ni() and from +lnet_startup_lndni() in case there is a failure starting the NI. +