[Beowulf] fast interconnects

[Patrick Geoffray]

Hi Gilad,

Gilad Shainer wrote:
>>a) There's likely to be 10Gbps ethernet over ordinary cat 5e/6 cabling
> 
> soon. (Solarflare is one company working on it)
> 
> It is not a surprise, as you can run InfiniBand on cat 6 cables today.
> There are several solution in the market that make it happened.  

You can run anything on a shoe string between two yogurt cups today, but 
it does not mean it's a good idea :-)

All Cat6 solutions have the same problems:

* overhead: KeyEye's transceivers for 10GigE (KX1001) adds 150ns. That's
0.3us for a point-to-point link if you assume it's per chip, twice that
with a switch in between. The transceiver for IB-SDR (KX1003) adds 
200ns, that's 0.4 us per link, 0.8 us with one hop.
* power: between 3.5 and 3.7 watts per chip. That's more than the rest
of the NIC consumption. On a 256-port switch, that's almost an extra 1KW 
to cool.
* length: 25-30 meters, not much more than CX4. Granted, it would
certainly be a bit thinner than a 20 meters "firehose" CX4 cable, but it
does not help on the length side.

The only advantages I see are price of the cable (to be confirmed) and 
look-like-cat5 warm feelings. Both are very important, but not for 
technical reasons.

Patrick