[Beowulf] ethernet bonding performance comparison "802.3ad" vs Adaptive Load Balancing

[Eric Thibodeau]

Rahul Nabar wrote:
> I was experimenting with using channel bonding my twin eth ports to
> get a combined bandwidth of (close to) 2 Gbps. The two relevant modes
> were 4 (802.3ad) and 6 (alb=Adaptive Load Balancing). I was trying to
> compare performance for both.
>
> Before running any sophisticated tests by netperf etc. I just tried to
> copy a large file via scp and timed the two file-copies.
>
> Option1:
> from node1 to node2. Both nodes have their twin ports bonded together
> as bond0 with mode=4 (802.3ad).
>
> They are connected via a Dell PowerConnect 6248 switch. Configured the
> switch so that I have two LAG groups combining the two ports coming
> from the same node. LACP was turned on.
>
> Option2:
> from node3 to node4. Use mode=6 (alb=Adaptive Load Balancing) No
> special switch config. No LAG. No LACP.
>
> Result: For a 4GB file-transfer. Both modes took the same time; approx
> 1min26 sec.
>
> These results are very mystifying to me. I was expecting mode4
> (802.3ad ) to be almost twice as fast since it is the only mode which
> truly aggregates the twin channels. It ought to be the only one
> effective for a peer-to-peer communication (mode 6 would only help
> while talking with more than one peer)
>
> Any comments? Also the net file transfer speed seems way lower than
> what I'd expect from a close to 2 Gbps connect; even accounting for
> the protocol overheads. Do other people have some numbers for me from
> their systems?
>   
Well, apart from the fact that ssh is compressed and, as Digo pointed 
out and that 47 MB/sec is probably your HDD's transfer capacity as 
Shannon pointed out, also keep in mind your bus's capacity ( 
http://en.wikipedia.org/wiki/List_of_device_bandwidths is a nice list). 
So, unless you've got both NICs on PCI-E (or independant PCI channels, 
which I've only heard of in high-end Compaq servers with hotswap PCI 
interfaces)  you're saturating your bus.

Eric