****************************************** * Overview of Dynamic LNet Configuration * ****************************************** Original Authors: ================= Amir Shehata Contents 1. Introduction 1.1 Objectives 1.3 Motivation 1.2 Overview 2. YAML Interface 3. DLC API 4. LNet Backend Design 4.1 router.c 4.1.1 Enabling/Disabling Routing and Router Pools 4.1.2 Configuring Router Pools 4.2 api-ni.c 4.2.1 Dynamically Adding an NI 4.2.2 Dynamically Deleting an NI 4.2.3 Internal Functions =================== = 1. Introduction = =================== 1.1 Objectives ============== Dynamic LNet Configuration (DLC) strives to accomplish four objectives: . Allow changes to some LNet parameters to be dynamic so they can be altered without having to stop/start LNet. . Remove the limits on how many NI's or route's which can be configured. . Work towards replacing the LNet module parameters with user space scripts (i.e. /etc/rc.d and /etc/sysconfig/network-scripts) which operate much more like conventional network config systems on Linux. . Allow sysadmins to use a command line utility for making dynamic changes to the LNet configuration. 1.2 Motivation ============== DLC introduces a new shared library. The DLC C API Library is provided as part of the standard lustre installation. This library must be released under LGPL license to facilitate third-party development. This API will serve the following purposes: . Provide a funnel for all LNet configurations. This way there isn't multiple ways to configure the same thing . Provide a C API which configures specific parameters. This C API will translate into IOCTL calls to the kernel module. . Provide a YAML interface, which ensures that YAML parsing is done in one location. . Provide more detailed return values from the kernel presented in YAML format. . Provide a set of API calls to parse out the YAML error returns if need be. 1.3 Overview ============ DLC introduces a userspace library which exposes a set of APIs to: . configure/unconfigure LNet . add/delete/show routes . add/delete/show networks . set router buffers sizes . enable/disable routing . import YAML configuration . export configuration in YAML format . show peer credit information DLC API uses IOCTLs to communicate with the kernel. This has the following advantage: . No need to do any string parsing in the kernel . Bypass the 4K limit, since each command will be sent to the kernel separately. For example, if there are 100 routes to configure then this translates to 100 ioctls to the kernel. . Provide a more intuitive user facing interface. The user need not know how to talk to the kernel via ioctl. This will be achieved in two ways: a. The configuration can be presented in YAML syntax, as shown below b. A more traditional C API will be exported, Each translates to exactly one ioctl to the kernel module. ===================== = 2. YAML Interface = ===================== The DLC API provides functions to configure specific parameters together with higher level functions which accept a YAML formatted configuration and configure all parameters described therein. To aid in parsing and building YAML stings a third party library, libyaml, is used. lnet/utils/cyaml/cyaml.c is written to interface with the libyaml library. When parsing a yaml string or file, it builds an internal tree representation of the YAML string. lnet/utils/cyaml/cyaml.c then provides functions to extract data from the tree. It also provides functions to build a YAML tree, which can then be printed textually. The YAML API is used by the DLC API to parse and extract YAML configuration, and to build a YAML representation of the success or error status per DLC API. ============== = 3. DLC API = ============== Refer to: lnet/utils/lnetconfig/liblnetconfig.h ========================== = 4. LNet Backend Design = ========================== 4.1 lnet/lnet/router.c ======================= In router.c the following three functions are used to add, delete and get a route. . lnet_add_route() : adds a new route entry. . lnet_del_route() : removes an existing routing entry. . lnet_get_route() : retrieves information on an existing route. These functions will be called from the ioctl handler. The Router buffer pools will be controlled via the following APIs: . lnet_rtrpools_enable() : enable routing and allocate router buffers . lnet_rtrpools_disable() : disable routing and free router buffers . lnet_rtrpools_adjust() : configure the number of router buffer pools 4.1.1 Enabling/Disabling Routing and Router Pools ================================================= lnet_rtrpools_enable() lnet_rtrpools_disable() These functions will explicitly enable and disable rtrpools respectively. If the rtrpools are being enabled then the default values of the tiny, small and large buffers are used. #define LNET_NRB_TINY (LNET_NRB_TINY_MIN * 4) #define LNET_NRB_SMALL (LNET_NRB_SMALL_MIN * 4) #define LNET_NRB_LARGE (LNET_NRB_LARGE_MIN * 4) 4.1.2 Configuring Router Pools ============================== lnet_rtrpools_adjust() This API will set the tiny, small and large buffers. rtrpools must be enabled prior to calling this API. If not enabled, this API is a no-op. Below is the behavior when adjusting the router pools: . A value of 0 indicates that the default values (listed above) are used. . Each pool size will be run through the current sizing algorithm to ensure the given values are not below the minimum standards. As such, a user could get the defaults for all three pools by just passing in "0 0 0" for pool sizes. . If a pool size is being made bigger, add the new buffers to the pool checking if there are any queued messages waiting for a buffer on that pool. If so, schedule those messages to be sent with the new buffer. . If a pool size is being made smaller, change the maximum value for the number of buffers on the pool only. When the system returns buffers to the pool, it will see there are "excess" buffers on the pool and will discard the buffer rather than return it to the pool. 4.2 lnet/lnet/api-ni.c ======================= The file api-ni.c contains the interface routines for working with network interfaces. Four new routines will be added to this file to support this feature. lnet_dyn_add_ni() : adds an NI dynamically lnet_dyn_del_ni() : deletes an NI dynamically lnet_startup_lndni() : starts a single LND. lnet_shutdown_lndni() : shuts down a single LND. 4.2.1 Dynamically Adding an NI ============================== lnet_dyn_add_ni() This function is called to startup a network passed in through userspace as a string. This function verifies that only one network is passed in. If more than one network is passed in, it fails with -EINVAL error number. Peer timeout, peer credits, and credits are passed in as well, since DLC makes these tunable parameters configurable per network interface. This function does the following: . Parses the network string passed in. . Initializes a new ping info structure in preparation of adding the Network Interface. . Starts up the NI . Updates the global ping info. . If there is any failure in any of these steps, appropriate cleanup action is taken. 4.2.2 Dynamically Deleting an NI ================================ lnet_dyn_del_ni() This function does the following: . Allocates a new ping info for the number of networks without the one being removed. . Shuts down the NI . Updates the global ping info appropriately. 4.2.3 Internal Functions ======================== lnet_startup_lndni() The code was restructured to breakup the function lnet_startup_lndnis() into two functions: lnet_startup_lndni() and lnet_startup_lndnis(). This way the new lnet_dyn_add_ni() can call lnet_startup_lndni() to startup one NI, and on standard startup lnet_startup_lndnis() can start up all NIs configured in the module parameters, by calling lnet_startup_lndni() repeatedly. lnet_shutdown_lndni() This function was introduced to shutdown only one NI. It coexists with the original lnet_shutdown_lndnis() which shuts down all NIs, which is needed when LNet is being unconfigured. lnet_shutdown_lndni() is called from lnet_dyn_del_ni() and from lnet_startup_lndni() in case there is a failure starting the NI.