[Beowulf] torus versus (fat) tree topologies

[Michael T. Prinkey]

On Fri, 12 Nov 2004, Mark Hahn wrote:

> > torus completely and, instead, added a second gigabit network using
> > "cheap" gigabit switches to handle I/O, logins, etc. and allow the
> > "expensive" gigabit network to handle the parallel traffic.
> 
> I'm curious about this: what about the expensive switch is worth
> its premium over cheap switches?  you probably don't use layer3
> or even management features, so perhaps latency or port-count?
> 
> I'd be interested if you had some numbers about latency of 
> various GE switch alternatives.  (feel free to post!)
> 
> 

The "cheap" and "expensive" switches are 24-port gigabit netgears (forget
the model number) and 48-port HP 2848s respectively.  Latency is better
for the HPs (5.4 us), but the primary difference is backplane bandwidth
and trunking ability.  We will be connecting four of the HPs together
using a fifth and several (I forget the limit) trunked links from each.

We are going to be installing the system in the next two weeks, so I will
have more benchmark data then.  The types of jobs that will be run on this
system will tend to fit on individual 48-port switches, so I don't
anticipate that the trunks will be a big bottleneck.

Another reason for the dual networks is to test MPI/Gamma.  When running,
MPI/Gamma "claims" the interface it is using, so normal TCP traffic will
need an alternate route to the system.  Also, moving I/O and X (Graphical
FLUENT) traffic off onto another LAN makes sense...especially considering
the incremental cost of adding it.

Mike Prinkey
Aeolus Research, Inc.