32-port gigabit switch

[Mark Hahn]

> I'm in the early stages of building a 32-node beowulf
> cluster, and I'm looking for a really _low-cost_
> 32-port (or more) gigabit switch for the LAN interconnection.

that begs to ask: *how* low-cost?

> I was wondering if I could use e.g. Foundry EdgeIron
> model 24G. But in that case it would be necessary to use
> two of them because, as the name implies, it's a 24-port model.

most vendors have fairly cheap 24pt gige switches now.

> I was told about using the 'flat neighborhood network'
> topology as a cheap alternative, using three 24G switches and two
> NICs on each node, but I have no idea if it's a good solution or not,

it's cool in the same way hash tables are cool, I think ;)

> It's important to mention that the interconnect structure I'm looking for
> should be easy to scale, up to maybe 128 compute nodes or so,
> maintaining full bisection bandwidth. Oh, yeah, we have a very limited
> budget, so at the present moment, Myrinet is out of question.

here's something we've been talking about building:

take 64 nodes, think of them as an 8x8 grid.  
take 16 switches, think of them as 8 "row" switches and 8 "col" switches.
plug each node into its row/col switches (assuming 2 gige ports/node).
plug each row switch into its 7 peer row switches.
plug each col switch into its 7 peer col switches.

any same-row or same-col link goes through a single switch;
other links go through two switches.  without the switch-switch links,
you'd have to route through a node (which not be a problem, donno).

bisection bandwidth is quite good; latency should be reasonable.
(though it's interesting that cheap gige switches seem to use 8-way 
building blocks, and the latency for a 3x8=24pt switch is quite a bit 
higher than a simple 8pt.)  unfortunately, vendors seem to be skipping
the 16pt model, so the basic scheme above uses 15 or 24 ports :(
then again, doing 2x aggregation on interswitch links might be a nice tweak,
and gets rid of all but two of those unsightly empty ports.

scaling to other lattices isn't hard; even the simple 2d scheme gets you 
to 12x12 with 24pt switches.

for fun this weekend, rewire the 8x8 above as 4x4x4 ;)