From 768c4366f27a56dccaec61b429f9c7a878c9f8ed Mon Sep 17 00:00:00 2001 From: Linda Bebernes Date: Thu, 19 Sep 2013 11:58:29 -0700 Subject: [PATCH] LUDOC-157 lnet: Add priority parameter to routes for LNet Change LU-2934 added a new LNet route parameter called priority. This change is to document the route priority parameter in the manual. Signed-off-by: Doug Oucharek Change-Id: I4cff3a9caf5ce3d1ef6285f4663f8cb31dba9cfd Reviewed-on: http://review.whamcloud.com/7428 Reviewed-by: Richard Henwood Tested-by: Hudson --- ConfigurationFilesModuleParameters.xml | 8 +++++--- ConfiguringLNET.xml | 3 ++- ManagingLNET.xml | 3 +++ 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/ConfigurationFilesModuleParameters.xml b/ConfigurationFilesModuleParameters.xml index c50db0e..ce0d9f3 100644 --- a/ConfigurationFilesModuleParameters.xml +++ b/ConfigurationFilesModuleParameters.xml @@ -160,10 +160,11 @@ routes ("") This is a string that lists networks and the NIDs of routers that forward to them. It has the following syntax (<w> is one or more whitespace characters): <routes> :== <route>{ ; <route> } -<route> :== [<net>[<w><hopcount>]<w><nid>{<w><nid>} +<route> :== [<net>[<w><hopcount>]<w><nid>[:<priority>]{<w><nid>[:<priority>]} + Note: the priority parameter was added in release 2.5. So a node on the network tcp1 that needs to go through a router to get to the Elan network: options lnet networks=tcp1 routes="elan 1 192.168.2.2@tcpA" - The hopcount is used to help choose the best path between multiply-routed configurations. + The hopcount and priority numbers are used to help choose the best path between multiply-routed configurations. A simple but powerful expansion syntax is provided, both for target networks and router NIDs as follows. <expansion> :== "[" <entry> { "," <entry> } "]" <entry> :== <numeric range> | <non-numeric item> @@ -171,7 +172,8 @@ routes ("") The expansion is a list enclosed in square brackets. Numeric items in the list may be a single number, a contiguous range of numbers, or a strided range of numbers. For example, routes="elan 192.168.1.[22-24]@tcp" says that network elan0 is adjacent (hopcount defaults to 1); and is accessible via 3 routers on the tcp0 network (192.168.1.22@tcp, 192.168.1.23@tcp and 192.168.1.24@tcp). routes="[tcp,vib] 2 [8-14/2]@elan" says that 2 networks (tcp0 and vib0) are accessible through 4 routers (8@elan, 10@elan, 12@elan and 14@elan). The hopcount of 2 means that traffic to both these networks will be traversed 2 routers - first one of the routers specified in this entry, then one more. Duplicate entries, entries that route to a local network, and entries that specify routers on a non-local network are ignored. - Equivalent entries are resolved in favor of the route with the shorter hopcount. The hopcount, if omitted, defaults to 1 (the remote network is adjacent). + Prior to release 2.5, a conflict between equivalent entries was resolved in favor of the route with the shorter hopcount. The hopcount, if omitted, defaults to 1 (the remote network is adjacent).. + Since 2.5, equivalent entries are resolved in favor of the route with the lowest priority number or shorter hopcount if the priorities are equal. The priority, if omitted, defaults to 0. The hopcount, if omitted, defaults to 1 (the remote network is adjacent). It is an error to specify routes to the same destination with routers on different local networks. If the target network string contains no expansions, then the hopcount defaults to 1 and may be omitted (that is, the remote network is adjacent). In practice, this is true for most multi-network configurations. It is an error to specify an inconsistent hop count for a given target network. This is why an explicit hopcount is required if the target network string specifies more than one network. diff --git a/ConfiguringLNET.xml b/ConfiguringLNET.xml index cabd7ed..c7f57a1 100644 --- a/ConfiguringLNET.xml +++ b/ConfiguringLNET.xml @@ -90,7 +90,8 @@ options lnet networks=tcp0(eth1) Depending on the network design, it may be necessary to specify explicit interfaces. To explicitly specify that interface eth2 be used for network tcp0 and eth3 be used for tcp1 , use this entry: options lnet networks=tcp0(eth2),tcp1(eth3) - When more than one interface is available during the network setup, Lustre chooses the best route based on the hop count. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to another interface, even if multiple interfaces are available on the same node. + When more than one interface is available during the network setup, Lustre chooses the best route based on the hopcount. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to another interface, even if multiple interfaces are available on the same node. + Since version 2.5 of the Lustre software, the route priority is also used to decide which interface to use. The priority has precendence over hopcount so the route with the lower priority number will be selected regardless of what the hopcounts are. LNET lines in lustre.conf are only used by the local node to determine what to call its interfaces. They are not used for routing decisions. diff --git a/ManagingLNET.xml b/ManagingLNET.xml index a6f3516..4d4d380 100644 --- a/ManagingLNET.xml +++ b/ManagingLNET.xml @@ -99,6 +99,9 @@ To remove all Lustre modules, run: A Lustre node always uses the same local NID to communicate with a given peer NID. The criteria used to determine the local NID are: + + Lowest route priority number (lower number, higher priority). + Fewest hops (to minimize routing), and -- 1.8.3.1