[Beowulf] Infiniband modular switches

[Mark Hahn]

> Static routing is the best approach if your pattern is known. In other

sure, but how often is the pattern actually known?  I mean in general:
aren't most clusters used for multiple, shifting purposes?

> There are some vendors that uses only the 24 port switches to build very
> large scale clusters - 3000 nodes and above, without any
> oversubscription, and they find it more cost effective. Using single

so the switch fabric would be a 'leaf' layer with 12 up and 12 down,
and a top layer with 24 down, right?  so 3000 nodes means 250 leaves
and 125 tops, 9000 total ports so 4500 cables.

> enclosures is easier, but the cables are not expensive and you can use
> the smaller components.

in federated networks, I think cables wind up being 15-20% of the network
price.  for instance, if we take the simplest possible approach, and equip
this 3000-node cluster with a non-blocking federated fabric (assuming
just sdr) from colfax's current price list:

subtot	unit	n	what
375000	125	3000	ib nic
117000	39	3000	1m host ib cables
148500	99	1500	8m leaf-top ib cables
900000	2400	375	24pt unman switch
1540500 total (cable 17%)

I'm still confused about IB pricing, since the street price of nics,
cables and switches are dramatically more expensive than colfax.
(to the paranoid, colfax would appear to be a mellanox shell company...)

for completeness, here's the same bom with "normal" public prices:

subtot	unit	n	what
2100000	700	3000	ib nic
330000	110	3000	1m host ib cables
330000	220	1500	8m leaf-top ib cables
1500000	4000	375	24pt unman switch
4260000 total (cable 15%)

interestingly, if nodes were about 3700 apiece (about what you'd expect
for intel dual-socket quad-core 2G/core), the interconnect winds up being 
28% of the cost.