[Beowulf] IB Cluster Configuration Options

[Mark Hahn]

> As a means to reduce our cost the suggestion was raised to cascade 4 9024 
> switches, or reducing the node count, Opteron clock rate, and memory. Just 
> wondering how severe the hop penalty might be in the case of cascading 
> switches?

IB switch vendors claim quite low per-hop latencies.  of course, the real 
problem is that the base of a non-fat switching tree is highly contended.
(the voltaire 9024 spec says 140 ns, for instance).

the question is: how much are you willing to pay for good
bisection-bandwidth?  personally, I cannot imagine buying IB except for 
extremely bandwidth-intensive apps, in which case, you probably want 
a non-blocking (full bisection) switching network.  the 9024 doesn't 
seem to be offered with all 12x ports, which is kind of odd, since 
a classic fat tree would be most natural with 8 12x ports at the top.
but a set of 5 9024-12 switches would get you a pretty well-connected 
set of 60 nodes.  a hypercube with express links would use 8 switches
and have great bandwidth and at most 3 switches on any path (I think!).

maybe you should consider alternatives (Myri 10G seems extremely attractive,
for instance).

regards, mark hahn.