1 <?xml version='1.0' encoding='UTF-8'?>
2 <chapter xmlns="http://docbook.org/ns/docbook"
3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en-US"
4 xml:id="settingupbonding">
5 <title xml:id="settingupbonding.title">Setting Up Network Interface Bonding</title>
6 <para>This chapter describes how to use multiple network interfaces in parallel to increase bandwidth and/or redundancy. Topics include:</para>
10 <xref linkend="network_interface_bonding_overview"/>
15 <xref linkend="interface_bonding_requirements"/>
20 <xref linkend="bonding_module_parameters"/>
25 <xref linkend="setting_up_bonding"/>
30 <xref linkend="configuring_lustre_with_bonding"/>
35 <xref linkend="bonding_references"/>
40 <para>Using network interface bonding is optional.</para>
42 <section xml:id="network_interface_bonding_overview">
43 <title>Network Interface Bonding Overview</title>
44 <para>Bonding, also known as link aggregation, trunking and port trunking, is a method of aggregating multiple physical network links into a single logical link for increased bandwidth.</para>
45 <para>Several different types of bonding are available in the Linux distribution. All these
46 types are referred to as 'modes', and use the bonding kernel module.</para>
47 <para>Modes 0 to 3 allow load balancing and fault tolerance by using multiple interfaces. Mode 4 aggregates a group of interfaces into a single virtual interface where all members of the group share the same speed and duplex settings. This mode is described under IEEE spec 802.3ad, and it is referred to as either 'mode 4' or '802.3ad.'</para>
49 <section xml:id="interface_bonding_requirements">
50 <title>Requirements</title>
51 <para>The most basic requirement for successful bonding is that both endpoints of the connection must be capable of bonding. In a normal case, the non-server endpoint is a switch. (Two systems connected via crossover cables can also use bonding.) Any switch used must explicitly handle 802.3ad Dynamic Link Aggregation.</para>
52 <para>The kernel must also be configured with bonding. All supported Lustre kernels have bonding functionality. The network driver for the interfaces to be bonded must have the ethtool functionality to determine slave speed and duplex settings. All recent network drivers implement it.</para>
53 <para>To verify that your interface works with ethtool, run:</para>
54 <screen># which ethtool
59 Supported ports: [ TP MII ]
60 Supported link modes: 10baseT/Half 10baseT/Full
61 100baseT/Half 100baseT/Full
62 Supports auto-negotiation: Yes
63 Advertised link modes: 10baseT/Half 10baseT/Full
64 100baseT/Half 100baseT/Full
65 Advertised auto-negotiation: Yes
72 Supports Wake-on: pumbg
74 Current message level: 0x00000001 (1)
80 Supported ports: [ TP MII ]
81 Supported link modes: 10baseT/Half 10baseT/Full
82 100baseT/Half 100baseT/Full
83 Supports auto-negotiation: Yes
84 Advertised link modes: 10baseT/Half 10baseT/Full
85 100baseT/Half 100baseT/Full
86 Advertised auto-negotiation: Yes
93 Supports Wake-on: pumbg
95 Current message level: 0x00000007 (7)
97 To quickly check whether your kernel supports bonding, run:
98 # grep ifenslave /sbin/ifup
103 <section xml:id="bonding_module_parameters">
104 <title>Bonding Module Parameters</title>
105 <para>Bonding module parameters control various aspects of bonding.</para>
106 <para>Outgoing traffic is mapped across the slave interfaces according to the transmit hash
107 policy. We recommend that you set the <literal>xmit_hash_policy</literal> option to the
108 layer3+4 option for bonding. This policy uses upper layer protocol information if available to
109 generate the hash. This allows traffic to a particular network peer to span multiple slaves,
110 although a single connection does not span multiple slaves.</para>
111 <screen>$ xmit_hash_policy=layer3+4</screen>
112 <para>The <literal>miimon</literal> option enables users to monitor the link status. (The
113 parameter is a time interval in milliseconds.) It makes an interface failure transparent to
114 avoid serious network degradation during link failures. A reasonable default setting is 100
115 milliseconds; run:</para>
116 <screen>$ miimon=100</screen>
117 <para>For a busy network, increase the timeout.</para>
119 <section xml:id="setting_up_bonding">
120 <title>Setting Up Bonding</title>
121 <para>To set up bonding:</para>
124 <para>Create a virtual 'bond' interface by creating a configuration file:</para>
125 <screen># vi /etc/sysconfig/network-scripts/ifcfg-bond0</screen>
128 <para>Append the following lines to the file.</para>
130 IPADDR=192.168.10.79 # Use the free IP Address of your network
132 NETMASK=255.255.255.0
138 <para>Attach one or more slave interfaces to the bond interface. Modify the eth0 and eth1 configuration files (using a VI text editor).</para>
141 <para>Use the VI text editor to open the eth0 configuration file.</para>
142 <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth0</screen>
145 <para>Modify/append the eth0 file as follows:</para>
151 BOOTPROTO=none</screen>
154 <para>Use the VI text editor to open the eth1 configuration file.</para>
155 <screen># vi /etc/sysconfig/network-scripts/ifcfg-eth1</screen>
158 <para>Modify/append the eth1 file as follows:</para>
170 <para>Set up the bond interface and its options in <literal>/etc/modprobe.d/bond.conf</literal>. Start the slave interfaces by your normal network method.</para>
171 <screen># vi /etc/modprobe.d/bond.conf
175 <para>Append the following lines to the file.</para>
176 <screen>alias bond0 bonding
177 options bond0 mode=balance-alb miimon=100
181 <para>Load the bonding module.</para>
182 <screen># modprobe bonding
184 # ifenslave bond0 eth0 eth1
190 <para>Start/restart the slave interfaces (using your normal network method).</para>
192 <para>You must <literal>modprobe</literal> the bonding module for each bonded interface. If you wish to create bond0 and bond1, two entries in <literal>bond.conf</literal> file are required.</para>
194 <para>The examples below are from systems running Red Hat Enterprise Linux. For setup use:
195 <literal>/etc/sysconfig/networking-scripts/ifcfg-*</literal> The website referenced
196 below includes detailed instructions for other configuration methods, instructions to use
197 DHCP with bonding, and other setup details. We strongly recommend you use this
200 <link xl:href="https://wiki.linuxfoundation.org/networking/bonding">
201 http://www.linuxfoundation.org/networking/bonding</link></para>
204 <para>Check /proc/net/bonding to determine status on bonding. There should be a file there for each bond interface.</para>
205 <screen># cat /proc/net/bonding/bond0
206 Ethernet Channel Bonding Driver: v3.0.3 (March 23, 2006)
208 Bonding Mode: load balancing (round-robin)
210 MII Polling Interval (ms): 0
214 Slave Interface: eth0
216 Link Failure Count: 0
217 Permanent HW addr: 4c:00:10:ac:61:e0
219 Slave Interface: eth1
221 Link Failure Count: 0
222 Permanent HW addr: 00:14:2a:7c:40:1d
226 <para>Use ethtool or ifconfig to check the interface state. ifconfig lists the first bonded interface as 'bond0.'</para>
228 bond0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
229 inet addr:192.168.10.79 Bcast:192.168.10.255 \ Mask:255.255.255.0
230 inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
231 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
232 RX packets:3091 errors:0 dropped:0 overruns:0 frame:0
233 TX packets:880 errors:0 dropped:0 overruns:0 carrier:0
234 collisions:0 txqueuelen:0
235 RX bytes:314203 (306.8 KiB) TX bytes:129834 (126.7 KiB)
237 eth0 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
238 inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
239 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
240 RX packets:1581 errors:0 dropped:0 overruns:0 frame:0
241 TX packets:448 errors:0 dropped:0 overruns:0 carrier:0
242 collisions:0 txqueuelen:1000
243 RX bytes:162084 (158.2 KiB) TX bytes:67245 (65.6 KiB)
244 Interrupt:193 Base address:0x8c00
246 eth1 Link encap:Ethernet HWaddr 4C:00:10:AC:61:E0
247 inet6 addr: fe80::4e00:10ff:feac:61e0/64 Scope:Link
248 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
249 RX packets:1513 errors:0 dropped:0 overruns:0 frame:0
250 TX packets:444 errors:0 dropped:0 overruns:0 carrier:0
251 collisions:0 txqueuelen:1000
252 RX bytes:152299 (148.7 KiB) TX bytes:64517 (63.0 KiB)
253 Interrupt:185 Base address:0x6000
258 <title>Examples</title>
259 <para>This is an example showing <literal>bond.conf</literal> entries for bonding Ethernet interfaces <literal>eth1</literal> and <literal>eth2</literal> to <literal>bond0</literal>:</para>
260 <screen># cat /etc/modprobe.d/bond.conf
264 options bond0 mode=balance-alb miimon=100
266 # cat /etc/sysconfig/network-scripts/ifcfg-bond0
269 NETMASK=255.255.255.0
270 IPADDR=192.168.10.79 # (Assign here the IP of the bonded interface.)
275 # cat /etc/sysconfig/network-scripts/ifcfg-eth0
278 HWADDR=4c:00:10:ac:61:e0
287 <para>In the following example, the <literal>bond0</literal> interface is the master (MASTER) while <literal>eth0</literal> and <literal>eth1</literal> are slaves (SLAVE).</para>
289 <para>All slaves of <literal>bond0</literal> have the same MAC address (Hwaddr) - <literal>bond0</literal>. All modes, except TLB and ALB, have this MAC address. TLB and ALB require a unique MAC address for each slave.</para>
291 <screen>$ /sbin/ifconfig
293 bond0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
294 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
295 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
296 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
297 TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
298 collisions:0 txqueuelen:0
300 eth0Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
301 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
302 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
303 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
304 TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
305 collisions:0 txqueuelen:100
306 Interrupt:10 Base address:0x1080
308 eth1Link encap:EthernetHwaddr 00:C0:F0:1F:37:B4
309 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
310 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
311 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
312 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
313 collisions:0 txqueuelen:100
314 Interrupt:9 Base address:0x1400</screen>
317 <section xml:id="configuring_lustre_with_bonding">
318 <title>Configuring a Lustre File System with Bonding</title>
319 <para>The Lustre software uses the IP address of the bonded interfaces and requires no special
320 configuration. The bonded interface is treated as a regular TCP/IP interface. If needed,
321 specify <literal>bond0</literal> using the Lustre <literal>networks</literal> parameter in
322 <literal>/etc/modprobe</literal>.</para>
323 <screen>options lnet networks=tcp(bond0)</screen>
325 <section xml:id="bonding_references">
326 <title>Bonding References</title>
327 <para>We recommend the following bonding references:</para>
330 <para>In the Linux kernel source tree, see
331 <literal>documentation/networking/bonding.txt</literal></para>
334 <para><link xl:href="http://linux-ip.net/html/ether-bonding.html">
335 http://linux-ip.net/html/ether-bonding.html</link>.</para>
338 <para>Linux Foundation bonding website:
339 <link xl:href="https://www.linuxfoundation.org/networking/bonding">
340 https://www.linuxfoundation.org/networking/bonding</link>. This
341 is the most extensive reference and we highly recommend it. This
342 website includes explanations of more complicated setups, including
343 the use of DHCP with bonding.</para>
349 vim:expandtab:shiftwidth=2:tabstop=8: