[Beowulf] fast interconnects

[Gilad Shainer]

> -----Original Message-----
> From: Mark Hahn [mailto:hahn at physics.mcmaster.ca] 
> Sent: Monday, May 22, 2006 11:56 AM
> To: Gilad Shainer
> Cc: beowulf at beowulf.org
> Subject: RE: [Beowulf] fast interconnects
> 
> > > a) There's likely to be 10Gbps ethernet over ordinary cat 5e/6 
> > > cabling
> > soon. (Solarflare is one company working on it)
> > 
> > It is not a surprise, as you can run InfiniBand on cat 6 cables
today.
> > There are several solution in the market that make it happened.  
> 
> interesting, but of limited import, IMO.  cat6 will presumably be
cheap (and light and long) relative to current cx4 cables, but the real
question is when (whether?) the ethernet ecosystem will embrace
10gbase-t and make it relatively cheap.
> 
> in other words, IB-over-cat6 is nice, but it doesn't fundamentally
change the IB picture.  I still have questions about IB:
> 
> 	- isn't Mellanox still the sole source for IB chips, for both
> 	nics and switches?  this seems odd if it's a thriving ecosystem.
> 	no offense intended!  yes, I know quadrics/SGI/SCI/Myri are all
> 	also sole-source.  but compared to the eth world...
>

Mellanox is not the sole source for InfiniBand chips. There are other
vendors like Qlogic and IBM that have InfiniBand chips. As InfiniBand
expend more and more, I will not be surprise to see more vendors doing
InfiniBand ASIC.

>
> 	- price of nics has gone down significantly, close to the point
> 	of no resistance.  but afaikt, IB switching is still a big cost;
> 	are there prospects of IB switch fabrics getting anywhere close
> 	to gigabit costs?
>

Its all customers demand. The market pushes the price down and we see
the results already. Mellanox ASIC is $69 and adapter is $125 OEM price.
IB provides much higher performance solution then GigE and I think one
needs to look on the price/performance figures and decide on the best
solution, rather on the price itself. Saying that, we do see the price
of the switches going down, but I am not the right guy to speak ob the
behalf of the switch vendors..

>
>
> 	- IB people seem to be claiming quite nice latency figures 
> 	these days, but I don't really understand what changed to cause
> 	a ~7x reduction in latency.
>

In one word experience. In the early days, IB was not design for high
performance but for other reasons. But if you can get high BW and low
latency then why not... Mellanox increases the performance from
generation to generation, and the latency we see today from multiple
vendors are quite nice figures.. 

>
>	- IB seems to be squeezed between cheap 1G (which is dirt cheap
>	and The Right Choice for most clusters), and the onset of 10G 
>	(which could satisfy the subset of bandwidth-intense clusters.)

>	would there be a reason to choose IB if the eth world suddenly 
>	started	shipping 10gbase-t in volume with switches costing, say,
>	$1000/pt?
>

InfiniBand and even InfiniBand DRY is much cheaper today than 10GigE.
10GigE people need to solve problems that do not exist in IB, and the
10GigE solutions for those issues will results in more expensive NICs.
With IB going to sub micro latency and 120Gbs BW, the results one can
get with IB will be much higher then 10GigE.   

>
> regards, mark hahn.
>

Best regards,
Gilad.