[Beowulf] split traffic to two interfaces for two "subnets"
list-beowulf at onerussian.com
Wed May 17 20:39:55 PDT 2006
> My quick and dirty test was to do NFS reads from memory on the server (to rule out disk contention) to multiple
> clients. So I did this:
> o On client 1, 'tar cO $DATA | cat > /dev/null' ~4GB of data from the
> server (4GB being the amount of memory in the server) to /dev/null.
> This caches the data in the server's RAM. Do the same from client 2.
> o On clients 1 and 2, tar the same data to /dev/null on both clients
Are you sure did step 1 on a client? that would fill up the client's
buffer (if its RAM < server RAM, then only partially) with content from
the server, thus during step 2 it will first fetch from the buffer and
read the rest from the server...
The reason I am asking because I did more tests on bonding and had to
postpone till I get all the nodes on the new switch.
with mode 0 I couldn't reach good throughput. Tried NFS from the nodes
- the same story and it looked weird -- 1 netperf stream would occupy
all 1Gbps and then only at the end let 2nd netperf to work for the
few milliseconds, so 2nd netperf was getting close to 1 Mbps at the
I've read up more about mode 0 from kernel docs:
balance-rr: This mode is the only mode that will permit a single
TCP/IP connection to stripe traffic across multiple
interfaces. It is therefore the only mode that will allow a
single TCP/IP stream to utilize more than one interface's
worth of throughput.
As I understood it: it does split traffic through both interfaces,
but if there are 2 TCP/IP streams, then for some reason it didn't do
any ballancing, so 1st stream occupied both, saturated switches
buffer and only after those gets delivered to the destination
node#1, it allowed the 2nd stream to get to node#2. Thus at the end
total throughput was limited by a single downlink throughput.
Is it something wrong with my kernel setup or this is how it has to
I've tried _xor bonding. That beast worked wonderfully -- occupying
both interfaces to their capacity whenever mac's XORed properly (so
reaching around 220 MBps total), and ofcause all traffic went through
the single interface if both destination MACs XORed to the same
If I don't resolve the issue with _rr I am afraid I would need to
tune MAC addressed of NICs in the nodes so I have even split between
them for _xor balancing
=------------------------------ /v\ ----------------------------=
Keep in touch // \\ (yoh@|www.)onerussian.com
Yaroslav Halchenko /( )\ ICQ#: 60653192
Linux User ^^-^^ 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 189 bytes
Desc: not available
More information about the Beowulf