[Beowulf] fast interconnects

[Jim Lux]

At 02:14 PM 5/19/2006, Greg Lindahl wrote:
>On Fri, May 19, 2006 at 08:49:19AM -0700, Jim Lux wrote:
>
> > a) There's likely to be 10Gbps ethernet over ordinary cat 5e/6 cabling
> > soon. (Solarflare is one company working on it)
>
>They're all trying, but it's unclear if it will be cheap enough and
>have low enough of a bit error rate to be useful in large clusters.


Indeed, it WILL be cheap enough.  The whole point of it is to make it lower 
cost than the various specialty interconnects, particularly with respect to 
the cabling costs.  The basic idea is that by using somewhat more 
sophisticated equalization (for the cross talk issues and impedance 
matching issues) and FEC to deal with the wire error rate, you can get good 
performance.

As for bit error rates.. 10^-15 is the going in worst case, and 10^-18 is 
the typical design point.  A bit of a challenge to test the latter however 
(10^18 bits at 10^10 bits/sec takes 10^8 seconds)



> > b) There's some interesting work from companies like Fulcrum on very fast
> > switches using asynchronous logic.
>
>Yep, this was publically shown at the last SC.
>
> > They have an architecture that essentially routes with zero copy
> > inside the switch.
>
>This has been the way most people build their non-Ethernet switch
>chips for a long time. There's still one copy. Don Becker has a funny
>definition of "zero copy", he says that it means someone else does
>the copy. This meets Don's defintion.

You're right.. I guess "real" zero copy would be something like wormhole 
routing where the packet is never stored.. circuit switched based on the 
header.


> > c) for high speeds, circuit switching is where it's at, rather than 
> classic
> > store and forward packet switching.
>
>I haven't seen any modern switch doing circuit switching. Everyone uses
>cut-through packet switching for non-Ethernet switches.

This is for the future, at high levels of aggregation.


As soon as I find out that the presentations are up on the web, I'll post a 
link.



James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875