Channel bonding: working combinations ?

[Kris Boutilier]

 Hi. I was unable to get Donalds ifenslave patches working at all on a
tweaked Redhat 6.1 server. This wasn't a beowulf situation, but rather
just a need for a bigger pipe into an existing fileserver. I was using
kernel 2.2.12-32 from Redhat, with v1.18 of the EtherExpress Pro/100
drivers from Andrey Savochkin. Most times bringing up the bond0 device
resulted in a kernel hang. In my case I was able to resolve the
situation by using the QoS components included in the 2.2.x kernels
instead. Specifically I compiled in the TEQL queue and used that as the
outbound packet scheduler for the trunked network cards. On my switch
(HP2424m) I was able to configure the two ports as a single SA trunk to
distribute the inbound traffic. Not the optimum soloution, a FEC trunk
would have better handled my traffic, but the Intel FEC drivers for
Linux for those cards weren't available at the time I had to put this
together. Your mileage may vary, particularly if you need very low
latency trunking...

 To summarise what I did (and this most definitly a kludge):

 In my kernel compilation options, I specifically activated:
	CONFIG_NET_SCH_TEQL=y
	CONFIG_NET_QOS=y
 You may also need other options, I can't remember...

 Then I downloaded, compiled and installed the iproute2 user space tools
(used to be available at ftp://ftp.inr.ac.ru/ip-routing/).
 Following that, I created the following files in
/etc/sysconfig/network-scripts:

	[root at DrEvil network-scripts.old]# cat ifcfg-eth0
	DEVICE="eth0"
	USERCTL="no"
	ONBOOT="yes"
	MASTER="teql0"
	IPADDR="192.168.48.250"
	NETMASK="255.255.255.0"
	BOOTPROTO="none"
	MACADDR="009027DC8CBD"

	[root at DrEvil network-scripts.old]# cat ifcfg-eth1
	DEVICE="eth1"
	USERCTL="no"
	ONBOOT="yes"
	MASTER="teql0"
	IPADDR="192.168.48.250"
	NETMASK="255.255.255.0"
	BOOTPROTO="none"
	MACADDR="009027DC8CBD"

	[root at DrEvil network-scripts]# cat ifcfg-teql0
	DEVICE="teql0"
	ONBOOT="yes"
	IPADDR="192.168.48.250"
	NETMASK="255.255.255.0"
	BOOTPROTO="none"

Then I twiddled my /sbin/ifup script to handle the new MASTER variable,
thus (note that the MACADDR is provided in the original script):
	[root at DrEvil /sbin]# diff ifup-pre-teql ifup
	3a4,7
	> #
	> # >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Tweaked by Kris to support
teql0 bonding
	> #
	>
	110,122c114,132
	<
	<     ifconfig ${DEVICE} ${IPADDR} netmask ${NETMASK} broadcast
${BROADCAST}
	<     # don't re-add subnet route on 2.2 kernels, but add a
route
	<     # to a non-local subnet.
	<     # stupid hack, but it should work
	<     if [ "$ISALIAS" = no ] && [ -z "`route -n | sed "s/ .*//"
| grep ${NETWORK}`" ]; then
	<       route add -net ${NETWORK} netmask ${NETMASK} dev
${DEVICE}
	<     else
	<       route add -host ${IPADDR} ${DEVICE}
	<     fi
	<
	<     # this is broken! it's only here for compatibility with
old RH systems
	<     if [ "${GATEWAY}" != "" -a "${GATEWAY}" != "none" ]; then
	---
	>
	>     if [ -z "$MASTER" ]; then
	>      if [ "${DEVICE}" = "teql0" ]; then
	>       # if this is a teql device then the bondings must be
made
	>       # before it can be ifconfig'ed
	>       /sbin/tc qdisc add dev eth0 root teql0
	>       /sbin/tc qdisc add dev eth1 root teql0
	>      fi
	>      ifconfig ${DEVICE} ${IPADDR} netmask ${NETMASK} broadcast
${BROADCAST}
	>      # don't re-add subnet route on 2.2 kernels, but add a
route
	>      # to a non-local subnet.
	>      # stupid hack, but it should work
	>      if [ "$ISALIAS" = no ] && [ -z "`route -n | sed "s/ .*//"
| grep ${NETWORK}`" ]; then
	>       route add -net ${NETWORK} netmask ${NETMASK} dev
${DEVICE}
	>      else
	>       route add -host ${IPADDR} ${DEVICE}
	>      fi
	>      # this is broken! it's only here for compatibility with
old RH systems
	>      if [ "${GATEWAY}" != "" -a "${GATEWAY}" != "none" ]; then
	124c134
	<     fi
	---
	>      fi
	126c136
	<     . /etc/sysconfig/network
	---
	>      . /etc/sysconfig/network
	128c138
	<     if [ "${GATEWAY}" != "" ]; then
	---
	>      if [ "${GATEWAY}" != "" ]; then
	133a144,148
	>      fi
	>     else
	>      # this device is a slave device for the teql0 system...
	>      ifconfig ${DEVICE} ${IPADDR} netmask ${NETMASK} broadcast
${BROADCAST}
	>      route del -net ${NETWORK} netmask ${NETMASK} dev
${DEVICE}

 That worked like a charm for me, at least until the official Intel
drivers for FEC bonding came out... now I use them. Note that this
approach treats queue scheduling on both the switch and the host
seperately and thus any one connection into the host will only get, at
maximum one full NIC to itself, although outbound packets will be
distributed across interfaces. If you want to spread inbound packets
from one session across the interfaces, that would need to be performed
by your switch (thus my need for Intel Fast EtherChannel).

 For more general info, I'd suggest doing a search on Google for 'tc
teql load balancing'. Also look out for the 'Differentiated Services on
Linux' mailing list (http://icawww1.epfl.ch/diffserv/) where you'll find
lots of info on the QoS tools themselves.

 Good luck! 

k.

> -----Original Message-----
> From:	Pfenniger Daniel [SMTP:daniel.pfenniger at obs.unige.ch]
> Sent:	Sunday, January 21, 2001 11:59 PM
> To:	beowulf at beowulf.org
> Subject:	Channel bonding: working combinations ?
> 
> Hi!
> 
> I am trying to install channel bonding on our cluster, but I meet a 
> few problems that may interest people on the list. 
> 
> 
	{clip}

> I am sure this would be much appreciated by those wanting to bond
> their Beowulf. 
> 
>         Daniel Pfenniger
> 
> 
>