[Beowulf] Clos networking and myrinet

[Mack.Joseph at epamail.epa.gov]

Joseph Mack PhD, High Performance Computing & Scientific Visualisation
LMIT, Supporting the EPA Research Triangle Park, NC 919-541-0007 Federal
Infrastructure Contact-Ravi Nair 919-541-5467 - nair.ravi at epa.gov,
Federal Visualization  Contact - Joe Retzer, Ph.D. 919-541-4190 -
retzer.joseph at epa.gov

Patrick Geoffray <patrick at myri.com> wrote on 06/08/2005 03:41:52 PM:

> Hi Joe,

> >>From other reading I find that 3 layer Clos networks
> > are blocking but that 5 (and above) layers are non-blocking.
>
> What do you mean by blocking in this context?

When I used the example of machines exchanging
packets in pairs, Greg Lindahl asked if I ran a lot
of 2 node clusters :-)

I was thinking of the situation where you're modelling
in 3-D and at the end of the computation you
have barriers to exchange data with the cell (node)
above you, then to the left, then in front. You would
want each pair to have their own path for the exchange.
If one pair couldn't exchange because the path was
in use, then I call this blocking. In an MPI situation
then all nodes would have to wait for the last pair
rather than all sending their data synchronously.
Presumably in the blocked situation, the exchange
would take twice as long as in the unblocked
situation.

I drew up an 8 node 3 layer Clos with 6 4 port
switches and looked to see if any particular
pair could exchange packets, when all other nodes
were exchanging in pairs and I came up with
situations where pairs of nodes would have
to wait for another pair to finish their exchange.

For the same number of nodes in a 5 layer Clos
network, all pairwise transfers of packets
were unblocked.


> Hope it helps, even if it looks confusing.

I'm getting there

Thanks Joe