[Beowulf] fast interconnects

[Mark Hahn]

> > 	- isn't Mellanox still the sole source for IB chips, for both
> > 	nics and switches?  this seems odd if it's a thriving ecosystem.
> > 	no offense intended!  yes, I know quadrics/SGI/SCI/Myri are all
> > 	also sole-source.  but compared to the eth world...
> 
> Mellanox is not the sole source for InfiniBand chips. There are other
> vendors like Qlogic and IBM that have InfiniBand chips. As InfiniBand

hmm, I know various people have produced IB chips, but I haven't seen
any in actual use.  oh, wait, qlogic - you mean pathscale infinipath.
yes, certainly, though it's worth noting that the interesting part of 
their product specifically just hijacks IB for their own faster MPI
protocol ;)

not unlike octigabay/cray-xd1, come to think of it, and not really
_that_ far from the Horus folk using IB phy layers for their non-IB
fabric.

> > 	- price of nics has gone down significantly, close to the point
> > 	of no resistance.  but afaikt, IB switching is still a big cost;
> > 	are there prospects of IB switch fabrics getting anywhere close
> > 	to gigabit costs?
> 
> Its all customers demand. The market pushes the price down and we see
> the results already. Mellanox ASIC is $69 and adapter is $125 OEM price.

what puzzles me is that market-push-down has apparenly only driven the 
prices of nics down.  that's a bit silly, since surely shaving $10 of 
memory off a $100 nic hurts performance, especially when the majority
of the installation's cost is in switch+cable (at least 5x the nic cost,
no?)

> > 	- IB people seem to be claiming quite nice latency figures 
> > 	these days, but I don't really understand what changed to cause
> > 	a ~7x reduction in latency.
> 
> In one word experience. In the early days, IB was not design for high
> performance but for other reasons. But if you can get high BW and low
> latency then why not... Mellanox increases the performance from
> generation to generation, and the latency we see today from multiple
> vendors are quite nice figures.. 

I'm afraid that upon reading this response and thinking about it,
I know no more than before ;(

> 10GigE people need to solve problems that do not exist in IB, and the
> 10GigE solutions for those issues will results in more expensive NICs.

but it's silly to only focus on nic prices.  I fully expect a myrinet
10gbase-t nic to cost at least around $500.  that sounds like a lot more
than a memfree IB nic, but if IB switch ports cost more than 10gbase-t
ports, it may be irrelevant.  besides, I'd actually prefer to pay a bit
more for my nics - to put intelligence in the nic rather than the switch.