[Beowulf] torus versus (fat) tree topologies

[Mark Hahn]

> I looking for some general comments about the pros/cons of torus (2D/3D) 
> and (fat) tree network topologies, for HPC of course.

latency, of course (bandwidth is easy).

> My investigation thus far has led me to believe that one reason a torus 
> topology might be better is because it eliminates the need for a switch. 

what do you have against switches?  *someone* has to do the switching,
so why not dedicated hardware?  think of it this way: in an n-dim mesh, you
only get 2^n-1 outputs per node, and only half-ish of those will make
progress, so each hop gives you only a small amount of "routing
computation".  compare this to something like myrinet, where switching is
done by 16 or 32x crossbar units.  not only fewer hops, but think also of 
the wiring - you don't really want connectors for all those mesh links,
do you?  not to mention tranceivers for each  you save a lot of pain by 
putting most or all the switching into a dedicated box which can etch 
all those links into a nice, reliable board with simpler PHY's and 
higher branching factor at each hop.

>   On the other hand fat tree interconnects seem to dominate the 
> larger(est) clusters out there, why?

practical factors, I believe: latency and packaging.  bisection bandwidth
seems to appeal to a lot of people, but I'm not sure why.  n-dim tori
have good BB which scales with size (and dimension).  if you really had 
code which was mostly near-neighbor, it's pretty easy to see mesh-like
interconnect making sense - you'd have little need for much routing
intelligence.  most codes I see have very non-NN patterns (need nontrivial
routing) and often do really care about latency.

and don't forget that switch-vs-mesh is not an either-or: you can do
interesting things to combine them (FNN, grid-of-switches, etc.)

regards, Mark Hahn.