[Beowulf] bonding and bandwidth

[Martin Siegert]

Hi Jean-Marc,
On Tue, Jun 08, 2004 at 08:46:51AM +0200, Jean-Marc Larré wrote:

> I'm testing bonding.c with two gigabit ethernet links toward a HP 2848 
> switch. My kernel is 2.4.24 from kernel.org on RedHat 9.0
> modules.conf is like this :
> [root at node01 root]# cat /etc/modules.conf
> alias eth1 bcm5700
> alias eth2 bcm5700
> alias bond0 bonding
> options bond0 miimon=100 mode=balance-alb updelay=50000
> 
> My problem :
> I get a bandwidth around 900Mbit/s and not 1800Mbit/s with netperf or 
> iperf or NetPipe. Could you explain me why I'm not get 1800Mbit/s and 
> where is my problem ?

Several conditions must be fullfilled to get more than 1Gbps out of
channel-bonded GigE NICs:
Most importantly: you must use round-robin mode. Since that is the default
under Linux you should test without any options for the bonding driver
first, i.e., just delete the "options bond0 ..." line from modules.conf.
You must also ensure that your switches support this particular mode
of link-aggregation (trunking, ether-channel, channel bonding, etc.).
The easiest setup would connect the NICs that correspond to eth1 to
one switch and the NICs the correspond to eth2 to a different switch.
However, this setup is possible only if all machines on that network
have two NICs and can use this type of channel bonding. If that is
not the case things get more complicated, because you have to find
a switch that supports round-robin mode on link-aggregates.

For the record: here is the result from an AIX box with 4 GigE fibre NICs
to a Linux/dual Opteron box with two broadcom NICs through
a switch that does support round-robin on link aggregates:

# netperf -l 600 -H 192.168.24.3 -- -s 1048576 -S 1048576 -m 1048576
TCP STREAM TEST to 192.168.24.3
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

262142 262142 1048576    600.01   1979.28

This is using jumbo frames.

Cheers,
Martin

-- 
Martif Siegert
Manager, Research Services
WestGrid Site Manager
Academic Computing Services                        phone: (604) 291-4691
Simon Fraser University                            fax:   (604) 291-4242
Burnaby, British Columbia                          email: siegert at sfu.ca
Canada  V5A 1S6