[Beowulf] How to configure a cluster network

[Mark Hahn]

> If you  need 144 ports a single switch will be be more cost effective

you'd think so - larger switches let you factor out lots of separate
little power supplies, etc.  not to mention transforming lots of cables
into compact, reliable, cheap backplanes.  but I haven't seen chassis
switches actually wind up cheaper.

of course, IB hardware prices seem to be extremely fuzzy (heavily 
discounted from list).

> than a gaggle of 24 ports gathered together into a tangled hairy cable ball with
> 144 ports exposed.

once very nice thing about the leaf/trunk fat-tree approach is that your
switches can be distributed in node racks.  so any individual rack has 
a fairly managable bundle of cables coming out of it.

> Your point about "most people don't need" is important!   With large
> multi core, multiple socket systems external and internal bandwidth
> can be interesting to ponder.

that makes it sound like inter-node networks in general are doomed ;)
while cores-per-node is increasing, users love to increase cores-per-job.