[Beowulf] The move to gigabit - technical questions

[Robert G. Brown]

On Tue, 15 Mar 2005, Vincent Diepeveen wrote:

> At 05:41 PM 3/14/2005 -0500, Glen Gardner wrote:
> >Gigabit will be a little faster than 100Mbit on a small cluster, but not 
> >a lot.
> 
> What is 'not a lot'.
> 
> I would guess it's factor 10 faster in bandwidth?

(Maybe, you don't get QUITE 100% of the raw clock advantage in all
applications on all hardware, Vincent;-).  However, for most
applications on most hardware you >>should<< get a signficant advantage
-- 80-95% of 10x, or 8-9.5x.  Not a just "a little".

A really, really cheap switch might have problems with bisection
bandwidth and chop this down for simultaneous flat-out bidirectional
data streams, but relatively few parallel applications engage in
flat-out bidirectional communications.  Even if it does, your problem is
more likely to be with resource contention (e.g. two hosts trying to
talk to a third at the same time) than it is with actual bandwidth
oversubscription.  This is what Vincent is suggesting that you look into
(or let us look into:-) below.

If your particular usage pattern does create resource contention, then
you might well need to either hand-optimize the pattern to avoid
saturating your cheap hardware, create a network with cheap components
that effectively breaks up the pathological communications pattern
(which it sounds like is what you actually did) or buy better hardware
(either better gigE switches or a "real" HPC network).

However you shouldn't really trash gigE itself -- it isn't at fault and
your results aren't typical.

    rgb

> 
> >I ended up using 5 cheap gigabit switches to make a gigabit concentrator 
> >for my 12 node cluster.
> >It eliminated the tendency for the network to saturate under a heavy load.
> 
> Very interesting, can you post a connection scheme and routing table?
> 
> >It also let me use gigabit network cards in my I/O node and controlling 
> >node with a small improvement in file I/O.
> 
> Streaming i/o or random access?
> 
> cheapo disk arrays get what is it, 400MB/s handsdown or so?
> 
> that's raid5 readspeed, plenty security at a raid5 array.
> 
> >The compute nodes remaind with 100 Mbit to conserve power. The setup 
> >works rather nicely.
> 
> what type of software do you run at it,
> embarrassingly parallel software?
> 
> Vincent
> 
> >Glen
> >
> >Vincent Diepeveen wrote:
> >
> >>Good evening,
> >>
> >>It's interesting to investigate what gigabit can do for small home clusters.
> >>
> >>Any latency oriented approach is doomed to fail obviously at gigabit. But
> >>they're cheap. For 40 euro i see several getting offered already.
> >>
> >>First important question is of course how much system time those NIC's eat
> >>when fully loading their bandwidth.
> >>
> >>Example, i have an old dual k7 here with pci 2.2 (32 bits 33Mhz).
> >>Suppose i put a gigabit card in it.
> >>
> >>In say 6 messages a second i ship 8MB data at a time. Ship and send in turn.
> >>
> >>So it ships a packet of 8MB, then receives a packet of 8MB.
> >>
> >>Other than the cost of the thread to store the packet to RAM, does such a
> >>card in any way stop or block the cpu's which are 100% loaded with
> >>searching software (my chessprogram diep in this case)?
> >>
> >>What penalty other than that thread handling the message is there in terms
> >>of system time reduction to the 2 processes searching?
> >>
> >>Oh btw, i assume that gigabit can handle 48MB/s user data a second?
> >>
> >>Vincent
> >>
> >>_______________________________________________
> >>Beowulf mailing list, Beowulf at beowulf.org
> >>To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> >>
> >>  
> >>
> >
> >-- 
> >Glen E. Gardner, Jr.
> >AA8C
> >AMSAT MEMBER 10593
> >Glen.Gardner at verizon.net
> >
> >
> >http://members.bellatlantic.net/~vze24qhw/index.html
> >
> >
> >
> >
> >
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu