[Beowulf] torus versus (fat) tree topologies

[Patrick Geoffray]

Hi Joachim,

It was good to see you at SC.

Joachim Worringen wrote:
> Yes, a feature of the NIC, plus: the torus doesn't affect the latency 
> significantly. In fact, the cost of a hop in an SCI-based torus, like an 
> other reasonable modern torus, is only a low number of nanoseconds (I'd 
> have to look it up right now for an exact number which I can not do from 
> here; I think there's also such a number in a recent publication on 
> BlueGene). A hop in a router node has about the same delay (as it is has 
> a similar task).

Going trough a crossbar cost about 100-150ns these days. I would expect 
the cost of one hop to be roughly the same in a torus. Now, the 
interesting number is the number of hops for a given number of nodes. 
With a 32-ports crossbar, you have one hop for 32 nodes. For 1280 nodes, 
you can build a Clos topology of diameter 5, thus 5 hops. With a 3D 
Torus, you route on a hypercube, so you have one hop for 8 nodes. I let 
the pleasure to compute the number of hops for a 1280 nodes 3D torus to 
a volunteer :-)

So the fabric overhead will depend on the size of the machine, as always.

Another consideration is the cabling. With 3D torus, you have 6 links 
going to each node. If they are on a backplane like BlueGene, that's 
fine. However, if those are short fat copper cables, you are in for a 
good time...

Patrick
-- 

Patrick Geoffray
Myricom, Inc.
http://www.myri.com