[Beowulf] SC|05: will 10Ge beat IB?

[Jeffrey B. Layton]

Olli-Pekka Lehto wrote: 

> Chelsio's T210-CX4 TOE NIC has been around for about a year. Like the 
> Myricom product it utilizes the 10GBase-CX standard. It essentially 
> pushes 10GbE over InfiniBand cabling directly using the XAUI interface 
> (normally used for on-PCB communication on 10G devices).
>
> There are also several vendors which provide interface modules for 
> switches. Of course with 10Gbase-CX you lose the advantage of common 
> and relatively cheap UTP cabling and are limited to 50ft per link. It 
> seems to me to be a stopgap standard that will be overtaken by 
> 10Gbase-T in a few years time.
>
> I don't have a quote on the pricing but it should be competitive with 
> IB. If anyone insight on the economics and performance of 10GBase-CX 
> vs. IB in cluster applications I'd be interested to hear it.


I wrote an Interconnect survey article about a year ago for
ClusterWorld magazine. I'm in the midst of updating the
article and I have a keen interest in the 10 GigE market.

The Chelsio CX NIC is about $800. You correct in that it
uses the new "skinny" IB copper cables and connectors. I
don't know the bend radius of these new cables, but the old
IB cables are very sensitive to bend radius (i.e. make the
radius of a bend too small and the cables are toast).

Switches are another matter. About a year ago, they were
very expensive. The NIC/switch combination was about
$5,000-$6,000 a port from 8 ports up to 128 ports. Now,
there are some good 8-12 port 10 GigE switches. One is
from HP and one is from Fujitsu. The per port cost is about
$1,200. So for small systems (8-12) nodes, the overall cost
per port is about $2,000 (minus cable costs).

To get about 8-12 nodes, you either need to go with a Foundry,
Force10, or Extreme 10 GigE line card (i.e. $$$).

Quadrics was showing a new 96-port 10 GigE switch at SC05.
It uses the Fujitsu ASIC. They use 8 of the 12 ports in the line
card and then use the other 4 ports to connect the line cards
together (it's 2:1 over subscribed). They haven't announced
any prices, but I'm guessing the per port price to be a bit
more than the cost for the 8-12 port switch.

Another company is developing a new 10 GigE switch ASIC.
Fulcrum Microsystems is developing a new ASIC with great
performance (200 ns latency) and 24 ports. The general cost
for a 24 port ASIC is about $20 a port, so the overall switch
costs should be lower if the phys costs aren't too high and the
vendor don't decide to gouge :) It's pretty easy to make a true
fat tree with up to 288 ports.

So, the overall 10 GigE costs start at about $2,000 per port
for small systems and move up after that.

I talked to Myricom about their 10G solution. A ballpark
price for it is about $1,200 a port, but I don't know their 10G
switch prices yet, so it's REALLY a ballpark.

My general rule of thumb for IB is about $1,000-$1,400 a port
but the real costs depend upon the configuration and the details.

Hope this helps!

Jeff

P.S. The interconnect survey will be posted on ClusterMonkey.net
as soon as I can get around to finishing it :)