large clusters and topologies

[Patrick GEOFFRAY]

Steffen Persvold wrote:

> > > Any alternatives to Myrinet upwards of that?
> >
> > Not really. It depends.
> SCI is a great alterative if you need high network bandwidth.
> It is also highly scalable.

"highly scalable" again :-)

SCI does not seems "highly scalable", on the paper and in reality:
On the paper, it's a shared bus design and the switch available
provides 6 ports. A link support 400 MB/s full duplex, so you can
put 2 nodes per SCI bus if you want to saturate your PCI 64/66.
The aggregate bandwidth of the switch is 1.28 GB/s, but 6 ports at
2*400 MB/s = 4.7 GB/s, it's far to be a full cross-bar. You can
cascade up to 12 switchs using 2 expansion ports per switch (4
user ports per switch available in this case), that means you
cannot keep an interesting bissection between all of the nodes of
your cluster.
In reality, the biggest SCI cluster that i know is in Paderborn :
96 nodes. I have talked in Mannheim with people who worked on it,
and it seems to be working as 3 clusters of 32 nodes.

I can say i have a scalable cluster if i can add machines without
decreasing dramatically the communication performance between each
peer of nodes. The "dramatically" part is very fuzzy : if you need
10 KB/s of bandwidth between your nodes, OK, you will scale very
well with Ethernet 10, Fast Ethernet or SCI. If you need 200 MB/s,
that's dramatically a problem. :-)

Regards
 
Patrick Geoffray
---
Aerospatiale Matra - Sycomore
Universite Lyon I - RESAM
http://lhpca.univ-lyon1.fr