[Beowulf] Clos networking and myrinet

[Mack.Joseph at epamail.epa.gov]

Joseph Mack PhD, High Performance Computing & Scientific Visualisation
LMIT, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal
Infrastructure Contact-Ravi Nair 919-541-5467 - nair.ravi at epa.gov,
Federal Visualization  Contact - Joe Retzer, Ph.D. 919-541-4190 -
retzer.joseph at epa.gov

I've just found out about Clos networks and realised that
myrinet uses Clos networks. From reading

http://www.beowulf.org/archive/2002-June/007458.html

I see that the diameter of the Clos network is log(n).
>From other reading I find that 3 layer Clos networks
are blocking but that 5 (and above) layers are non-blocking.

The url above shows that Myrinet uses a 3 layer Clos
network for 24..128 nodes, but presumably the network
is blocking. In an mpi type beowulf running a scatter/gather,
if one node is blocked, then all nodes have to wait.
Is this OK or am I missing something?

With large beowulfs, why is the diameter of the network
increased from 3 to 5 to 7...? It would be cheaper to
keep the networking at 3 layers. What falls apart
that you need to increase the diameter of the network?

Sorry if this is all a FAQ. I've been reading for about
2 weeks now and don't understand Clos networking
yet.

Thanks Joe