HOWTO discriminate switches?

[Richard Walsh]

Hey Patrick. here's what I was thinking:

>Hi Richard,
>
>On Mon, 17 Sep 2001, Richard Walsh wrote:
> 
>> > Another alternative which starts to make sense as the switching cost
>> > of your system rises as a percentage of total cost is a switchless
>> > system in the form of a 1D or 2D torus using SCI cards which have
>> > extremely low latency and high bandwdth. Take a look at the cluster
>> > at the University of Delaware based on this design. If you do the
>> > math, I think you will find that for larger clusters which would
>> > require multiple layers of switches or a "mega-switch" to build
>> > in reasonable bandwidth, the SCI torus design is no more costly and
>
>80 nodes would fit easily on only one Myrinet switch, based on a 
>128 ports enclosure and 10 line cards (with place to expand to 128 
>ports by adding line cards). This solution provides full
>bandwidth bissection at 2 Gb/s full-duplex (Clos topology). 
>Hard to do it with a Torus design.

Without trying to be a salesman for SCI, they claim their 66/64 PCI
cards to be 1.33 G_BYTES (not bits) per second bi-directionally with a
1.46 microsecond latency. The is point-to-point and is reduced as
the distance to travel through the torus increases, but is potentially
faster that Myrnet.  Those with actual SCI experience are free to 
add or subtract value from this point.

>
>Large switched cluster are no a problem if you know how to build
>inexpensive large switch. The real math is that it cost the same 
>thing that a switchless topology like SCI. 
>FYI, the total price of the interconnect would be $140K with Myrinet,
>including NIC, switch, cables and software (public price from the web).
>

I do not know about the _REAL_ math, buy an anlysis that I did for
a 512 node system a year ago comparing switched versus non-switched
topologies based on SCI cards showed the SCI (cards only) configured 
system to be cheaper than Myrnet or Gigabt switched alternatives
(cards and switches). Baseline performance of the network had to be 
close to that of the Cray T3E.

>> > are equal to or better than some custom engineered networks from
>> > vendors of SMP and MPP systems (Cray, Sun, SGI).
>
>What are these "custom engineered networks" ?

I was refering to the interconnect on the Cray T3E.

Thanks for the Myrnet information,

rbw