[Beowulf] bonding and bandwidth
siegert at sfu.ca
Wed Jun 9 13:10:41 PDT 2004
On Tue, Jun 08, 2004 at 08:46:51AM +0200, Jean-Marc Larré wrote:
> I'm testing bonding.c with two gigabit ethernet links toward a HP 2848
> switch. My kernel is 2.4.24 from kernel.org on RedHat 9.0
> modules.conf is like this :
> [root at node01 root]# cat /etc/modules.conf
> alias eth1 bcm5700
> alias eth2 bcm5700
> alias bond0 bonding
> options bond0 miimon=100 mode=balance-alb updelay=50000
> My problem :
> I get a bandwidth around 900Mbit/s and not 1800Mbit/s with netperf or
> iperf or NetPipe. Could you explain me why I'm not get 1800Mbit/s and
> where is my problem ?
Several conditions must be fullfilled to get more than 1Gbps out of
channel-bonded GigE NICs:
Most importantly: you must use round-robin mode. Since that is the default
under Linux you should test without any options for the bonding driver
first, i.e., just delete the "options bond0 ..." line from modules.conf.
You must also ensure that your switches support this particular mode
of link-aggregation (trunking, ether-channel, channel bonding, etc.).
The easiest setup would connect the NICs that correspond to eth1 to
one switch and the NICs the correspond to eth2 to a different switch.
However, this setup is possible only if all machines on that network
have two NICs and can use this type of channel bonding. If that is
not the case things get more complicated, because you have to find
a switch that supports round-robin mode on link-aggregates.
For the record: here is the result from an AIX box with 4 GigE fibre NICs
to a Linux/dual Opteron box with two broadcom NICs through
a switch that does support round-robin on link aggregates:
# netperf -l 600 -H 192.168.24.3 -- -s 1048576 -S 1048576 -m 1048576
TCP STREAM TEST to 192.168.24.3
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
262142 262142 1048576 600.01 1979.28
This is using jumbo frames.
Manager, Research Services
WestGrid Site Manager
Academic Computing Services phone: (604) 291-4691
Simon Fraser University fax: (604) 291-4242
Burnaby, British Columbia email: siegert at sfu.ca
Canada V5A 1S6
More information about the Beowulf