From: Andreas Dilger Date: Thu, 19 Jan 2012 15:31:30 +0000 (-0700) Subject: LUDOC-37 kernel: use /etc/modprobe.d/lustre.conf X-Git-Tag: 2.2.0~23 X-Git-Url: https://git.whamcloud.com/gitweb?a=commitdiff_plain;h=refs%2Fchanges%2F88%2F1988%2F3;p=doc%2Fmanual.git LUDOC-37 kernel: use /etc/modprobe.d/lustre.conf Update the documentaion to store module parameters in the /etc/modprobe.d/lustre.conf file to make the newer distro modutils happy. Early versions of modprobe did not care what filenames were used in /etc/modprobe.d/, which lead to confusion if old versions of files were located there (e.g. Lustre.orig or Lustre~). Newer versions of modprobe (at least since FC12, but possibly earlier) require module parameter files to have a ".conf" suffix in order to be parsed. The /etc/modprobe.d/ directory has been supported since at least module-init-utils 3.2 (RHEL5) and may have been supported before that time. This means that all supported distros can use /etc/modprobe.d/lustre.conf for Lustre-specific parameters. Signed-off-by: Andreas Dilger Change-Id: Ic885f610b3d9b16eb211d4a2d971930f53647bec --- diff --git a/ConfigurationFilesModuleParameters.xml b/ConfigurationFilesModuleParameters.xml index a3adf0d..ba11c86 100644 --- a/ConfigurationFilesModuleParameters.xml +++ b/ConfigurationFilesModuleParameters.xml @@ -17,21 +17,12 @@ Introduction - LNET network hardware and routing are now configured via module parameters. Parameters should be specified in the /etc/modprobe.conf file, for example: - alias lustre llite -options lnet networks=tcp0,elan0 - The above option specifies that this node should use all the available TCP and Elan interfaces. + LNET network hardware and routing are now configured via module parameters. Parameters should be specified in the /etc/modprobe.d/lustre.conffile, for example: + options lnet networks=tcp0(eth2) + The above option specifies that this node should use the TCP protocol on the eth2 network interface. Module parameters are read when the module is first loaded. Type-specific LND modules (for instance, ksocklnd) are loaded automatically by the LNET module when LNET starts (typically upon modprobe ptlrpc). - Under Linux 2.6, LNET configuration parameters can be viewed under /sys/module/; generic and acceptor parameters under LNET, and LND-specific parameters under the name of the corresponding LND. - Under Linux 2.4, sysfs is not available, but the LND-specific parameters are accessible via equivalent paths under /proc. - Important: All old (pre v.1.4.6) Lustre configuration lines should be removed from the module configuration files and replaced with the following. Make sure that CONFIG_KMOD is set in your linux.config so LNET can load the following modules it needs. The basic module files are: - modprobe.conf (for Linux 2.6) - alias lustre llite -options lnet networks=tcp0,elan0 - modules.conf (for Linux 2.4) - alias lustre llite -options lnet networks=tcp0,elan0 - For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. (Unmarked parameters can only be set when LNET loads for the first time.) Changes to parameters marked with Wc only have effect when connections are established (existing connections are not affected by these changes.) + LNET configuration parameters can be viewed under /sys/module/lnet/parameters/, and LND-specific parameters under the name of the corresponding LND, for example /sys/module/ksocklnd/parameters/ for the socklnd (TCP) LND. + For the following parameters, default option settings are shown in parenthesis. Changes to parameters marked with a W affect running systems. Unmarked parameters can only be set when LNET loads for the first time. Changes to parameters marked with Wc only have effect when connections are established (existing connections are not affected by these changes.)
@@ -46,7 +37,7 @@ options lnet networks=tcp0,elan0</screen> <para>For a routed network, use the same 'routes' configuration everywhere. Nodes specified as routers automatically enable forwarding and any routes that are not relevant to a particular node are ignored. Keep a common configuration to guarantee that all nodes have consistent routing tables.</para> </listitem> <listitem> - <para>A separate <literal>modprobe.conf.lnet</literal> included from <literal>modprobe.conf</literal> makes distributing the configuration much easier.</para> + <para>A separate <literal>lustre.conf</literal> file makes distributing the configuration much easier.</para> </listitem> <listitem> <para>If you set <literal>config_on_load=1</literal>, LNET starts at <literal>modprobe</literal> time rather than waiting for Lustre to start. This ensures routers start working at module load time.</para> diff --git a/ConfiguringLNET.xml b/ConfiguringLNET.xml index 9b72087..5ddfa75 100644 --- a/ConfiguringLNET.xml +++ b/ConfiguringLNET.xml @@ -41,7 +41,7 @@ Overview of LNET Module Parameters LNET kernel module (lnet) parameters specify how LNET is to be configured to work with Lustre, including which NICs will be configured to work with Lustre and the routing to be used with Lustre. - Parameters for lnet are specified in the modprobe.conf or modules.conf file (depending on your Linux distribution) in one or more entries with the syntax: + Parameters for LNET can be specified in the /etc/modprobe.d/lustre.conf file. In some cases the parameters may have been stored in /etc/modprobe.conf, but this has been deprecated since before RHEL5 and SLES10, and having a separate /etc/modprobe.d/lustre.conf file simplifies administration and distribution of the Lustre networking configuration. This file contains one or more entries with the syntax: options lnet <parameter>=<parameter value> To specify the network interfaces that are to be used for Lustre, set either the networks parameter or the ip2nets parameter (only one of these parameters can be used at a time): @@ -82,7 +82,7 @@
<indexterm><primary>LNET</primary><secondary>module parameters</secondary></indexterm>Setting the LNET Module networks Parameter - If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the modprobe.conf file on the node that sets the LNET module networks parameter: + If a node has more than one network interface, you'll typically want to dedicate a specific interface to Lustre. You can do this by including an entry in the lustre.conf file on the node that sets the LNET module networks parameter: options lnet networks=<comma-separated list of networks> This example specifies that a Lustre node will use a TCP/IP interface and an InfiniBand interface: options lnet networks=tcp0(eth0),o2ib(ib0) @@ -92,7 +92,7 @@ options lnet networks=tcp0(eth2),tcp1(eth3) When more than one interface is available during the network setup, Lustre chooses the best route based on the hop count. Once the network connection is established, Lustre expects the network to stay connected. In a Lustre network, connections do not fail over to another interface, even if multiple interfaces are available on the same node. - LNET lines in modprobe.conf are only used by the local node to determine what to call its interfaces. They are not used for routing decisions. + LNET lines in lustre.conf are only used by the local node to determine what to call its interfaces. They are not used for routing decisions.
<indexterm><primary>configuing</primary><secondary>multihome</secondary></indexterm>Multihome Server Example @@ -114,16 +114,16 @@ To set the networks option for this example: - On each server, svr1 and svr2, include the following line in the modprobe.conf file: + On each server, svr1 and svr2, include the following line in the lustre.conf file: options lnet networks=tcp0(eth0),tcp1(eth1),o2ib - For TCP-only clients, the first available non-loopback IP interface is used for tcp0. Thus, TCP clients with only one interface do not need to have options defined in the modprobe.conf file. + For TCP-only clients, the first available non-loopback IP interface is used for tcp0. Thus, TCP clients with only one interface do not need to have options defined in the lustre.conf file. - On the InfiniBand clients, include the following line in the modprobe.conf file: + On the InfiniBand clients, include the following line in the lustre.conf file: options lnet networks=o2ib @@ -137,7 +137,7 @@
<indexterm><primary>LNET</primary><secondary>ip2nets</secondary></indexterm>Setting the LNET Module ip2nets Parameter - The ip2nets option is typically used when a single, universal modprobe.conf file is run on all servers and clients. Each node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses. + The ip2nets option is typically used when a single, universal lustre.conf file is run on all servers and clients. Each node identifies the locally available networks based on the listed IP address patterns that match the node's local IP addresses. Note that the IP address patterns listed in the ip2nets option are only used to identify the networks that an individual node should instantiate. They are not used by LNET for any other communications purpose. For the example below, the nodes in the network have these IP addresses: @@ -154,11 +154,11 @@ Infiniband clients have IP over Infiniband (o2ib) addresses 132.6.[2-3].2, .4, .6, .8. - The following entry is placed in the modprobe.conf file on each server and client: + The following entry is placed in the lustre.conf file on each server and client: options lnet 'ip2nets="tcp0(eth0) 192.168.0.[2,4]; \ tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"' Each entry in ip2nets is referred to as a 'rule'. - The order of LNET entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in modprobe.conf will be used. + The order of LNET entries is important when configuring servers. If a server node can be reached using more than one network, the first network specified in lustre.conf will be used. Because svr1 and svr2 match the first rule, LNET uses eth0 for tcp0 on those machines. (Although svr1 and svr2 also match the second rule, the first matching rule for a particular network is used). The [2-8/2] format indicates a range of 2-8 stepped by 2; that is 2,4,6,8. Thus, the clients at 132.6.3.5 will not find a matching o2ib network.
@@ -175,7 +175,7 @@ tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"' The number of LNET routers is not limited. Enough routers should be used to handle the required file serving bandwidth plus a 25 percent margin for headroom.
<indexterm><primary>LNET</primary><secondary>routing example</secondary></indexterm>Routing Example - On the clients, place the following entry in the modprobe.conf file + On the clients, place the following entry in the lustre.conf file lnet networks="tcp" routes="o2ib0 192.168.0.[1-8]@tcp0" On the router nodes, use: lnet networks="tcp o2ib" forwarding=enabled @@ -193,7 +193,7 @@ lctl network configure
<indexterm><primary>LNET</primary><secondary>route checker</secondary></indexterm>Configuring the Router Checker In a Lustre configuration in which different types of networks, such as a TCP/IP network and an Infiniband network, are connected by routers, a router checker can be run on the clients and servers in the routed configuration to monitor the status of the routers. In a multi-hop routing configuration, router checkers can be configured on routers to monitor the health of their next-hop routers. - A router checker is configured by setting lnet parameters in modprobe.conf by including an entry in this form: + A router checker is configured by setting lnet parameters in lustre.conf by including an entry in this form: options lnet <router checker parameter>=<parameter value> The router checker parameters are: diff --git a/InstallingLustreFromSourceCode.xml b/InstallingLustreFromSourceCode.xml index 5382b16..3566083 100644 --- a/InstallingLustreFromSourceCode.xml +++ b/InstallingLustreFromSourceCode.xml @@ -340,7 +340,7 @@ lustre-ldiskfs-3.0.6-2.6.18_92.1.10.el5_lustre.1.6.6\ smp.x86_64.rpm - Add the following lines to the /etc/modprobe.conf file. + Add the following lines to the /etc/modprobe.d/lustre.conf file. options kmxlnd hosts=/etc/hosts.mxlnd options lnet networks=mx0(myri0),tcp0(eth0) diff --git a/LustreMaintenance.xml b/LustreMaintenance.xml index 4246d3d..12c28f1 100644 --- a/LustreMaintenance.xml +++ b/LustreMaintenance.xml @@ -205,7 +205,7 @@ Changing a Server NID To change a server NID: - Update the LNET configuration in the /etc/modprobe.conf file so the list of server NIDs (lctl list_nids) is correct. + Update the LNET configuration in the /etc/modprobe.d/lustre.conf file so the list of server NIDs (lctl list_nids) is correct. The lctl list_nids command indicates which network(s) are configured to work with Lustre. diff --git a/LustreTroubleshooting.xml b/LustreTroubleshooting.xml index d513b7c..d45fbc0 100644 --- a/LustreTroubleshooting.xml +++ b/LustreTroubleshooting.xml @@ -407,7 +407,7 @@ tail -30 /tmp/objects.{diskname} Start Lustre before starting any service that uses sunrpc. - Use a port other than 988 for Lustre. This is configured in /etc/modprobe.conf as an option to the LNET module. For example: + Use a port other than 988 for Lustre. This is configured in /etc/modprobe.d/lustre.conf as an option to the LNET module. For example: options lnet accept_port=988 diff --git a/LustreTuning.xml b/LustreTuning.xml index 57d216d..2afab4f 100644 --- a/LustreTuning.xml +++ b/LustreTuning.xml @@ -20,7 +20,7 @@ - Many options in Lustre are set by means of kernel module parameters. These parameters are contained in the modprobe.conf file. + Many options in Lustre are set by means of kernel module parameters. These parameters are contained in the /etc/modprobe.d/lustre.conf file.
diff --git a/ManagingLNET.xml b/ManagingLNET.xml index c754977..74e5eaf 100644 --- a/ManagingLNET.xml +++ b/ManagingLNET.xml @@ -110,12 +110,12 @@ To remove all Lustre modules, run:</para> <title><indexterm><primary>LNET</primary><secondary>Infiniband load balancing</secondary></indexterm>Load Balancing with InfiniBand A Lustre file system contains OSSs with two InfiniBand HCAs. Lustre clients have only one InfiniBand HCA using OFED Infiniband ''o2ib'' drivers. Load balancing between the HCAs on the OSS is accomplished through LNET.
- <indexterm><primary>LNET</primary><secondary>modprobe.conf</secondary></indexterm>Setting Up <literal>modprobe.conf</literal> for Load Balancing + <indexterm><primary>LNET</primary><secondary>lustre.conf</secondary></indexterm>Setting Up <literal>lustre.conf</literal> for Load Balancing To configure LNET for load balancing on clients and servers: - Set the modprobe.conf options. - Depending on your configuration, set modprobe.conf options as follows: + Set the lustre.conf options. + Depending on your configuration, set lustre.conf options as follows: Dual HCA OSS server diff --git a/SettingUpBonding.xml b/SettingUpBonding.xml index 6c08266..e93241b 100644 --- a/SettingUpBonding.xml +++ b/SettingUpBonding.xml @@ -158,8 +158,8 @@ BOOTPROTO=none - Set up the bond interface and its options in /etc/modprobe.conf. Start the slave interfaces by your normal network method. - # vi /etc/modprobe.conf + Set up the bond interface and its options in /etc/modprobe.d/bond.conf. Start the slave interfaces by your normal network method. + # vi /etc/modprobe.d/bond.conf @@ -180,7 +180,7 @@ options bond0 mode=balance-alb miimon=100 Start/restart the slave interfaces (using your normal network method). - You must modprobe the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in modprobe.conf are required. + You must modprobe the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in bond.conf file are required. The examples below are from RedHat systems. For setup use: /etc/sysconfig/networking-scripts/ifcfg-* The website referenced below includes detailed instructions for other configuration methods, instructions to use DHCP with bonding, and other setup details. We strongly recommend you use this website. http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding @@ -241,18 +241,12 @@ eth1 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
Examples - This is an example showing modprobe.conf entries for bonding Ethernet interfaces eth1 and eth2 to bond0: - # cat /etc/modprobe.conf + This is an example showing bond.conf entries for bonding Ethernet interfaces eth1 and eth2 to bond0: + # cat /etc/modprobe.d/bond.conf alias eth0 8139too -alias scsi_hostadapter sata_via -alias scsi_hostadapter1 usb-storage -alias snd-card-0 snd-via82xx -options snd-card-0 index=0 -options snd-via82xx index=0 +alias eth1 via-rhine alias bond0 bonding options bond0 mode=balance-alb miimon=100 -options lnet networks=tcp -alias eth1 via-rhine # cat /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0 diff --git a/SettingUpLustreSystem.xml b/SettingUpLustreSystem.xml index 41785fe..4829880 100644 --- a/SettingUpLustreSystem.xml +++ b/SettingUpLustreSystem.xml @@ -525,7 +525,7 @@ Lustre routers to connect the two networks. - Lustre networks and routing are configured and managed by specifying parameters to the Lustre Networking (lnet) module in /etc/modprobe.conf or /etc/modprobe.conf.local (depending on your Linux distribution). + Lustre networks and routing are configured and managed by specifying parameters to the Lustre Networking (lnet) module in /etc/modprobe.d/lustre.conf. To prepare to configure Lustre Networking, complete the following steps: @@ -544,7 +544,7 @@ To ease the setup of networks with complex network configurations, determine a cluster-wide module configuration. - For large clusters, you can configure the networking setup for all nodes by using a single, unified set of parameters in the modprobe.conf file on each node. Cluster-wide configuration is described in . + For large clusters, you can configure the networking setup for all nodes by using a single, unified set of parameters in the lustre.conf file on each node. Cluster-wide configuration is described in .