[Beowulf] fast interconnects
Shainer at mellanox.com
Tue May 23 17:20:12 PDT 2006
> > > - isn't Mellanox still the sole source for IB chips, for both
> > > nics and switches? this seems odd if it's a thriving ecosystem.
> > > no offense intended! yes, I know quadrics/SGI/SCI/Myri are all
> > > also sole-source. but compared to the eth world...
> > Mellanox is not the sole source for InfiniBand chips. There are
> > vendors like Qlogic and IBM that have InfiniBand chips. As
> hmm, I know various people have produced IB chips, but I haven't seen
any in actual use. oh, wait, qlogic - you mean pathscale infinipath.
> yes, certainly, though it's worth noting that the interesting part of
their product specifically just hijacks IB for their own faster MPI
> not unlike octigabay/cray-xd1, come to think of it, and not really
_that_ far from the Horus folk using IB phy layers for their non-IB
> > > - price of nics has gone down significantly, close to the point
> > > of no resistance. but afaikt, IB switching is still a big cost;
> > > are there prospects of IB switch fabrics getting anywhere close
> > > to gigabit costs?
> > Its all customers demand. The market pushes the price down and we
> > the results already. Mellanox ASIC is $69 and adapter is $125 OEM
> what puzzles me is that market-push-down has apparenly only driven the
prices of nics down. that's a bit silly, since surely shaving > $10 of
memory off a $100 nic hurts performance, especially when the majority of
the installation's cost is in switch+cable (at least > 5x the nic cost,
In any of the testing I know of, there was no performance difference
between Memfree and nics with memory. Shaving the nics memory does not
only same some $ but also increase the reliability of your cluster.
> > > - IB people seem to be claiming quite nice latency figures
> > > these days, but I don't really understand what changed to cause
> > > a ~7x reduction in latency.
> > In one word experience. In the early days, IB was not design for
> > performance but for other reasons. But if you can get high BW and
> > latency then why not... Mellanox increases the performance from
> > generation to generation, and the latency we see today from multiple
> > vendors are quite nice figures..
> I'm afraid that upon reading this response and thinking about it, I
know no more than before ;(
You asked on the how come the latency has improved that much. It is all
about optimizations in both HW and SW.... And more to come in the near
> > 10GigE people need to solve problems that do not exist in IB, and
> > 10GigE solutions for those issues will results in more expensive
> but it's silly to only focus on nic prices. I fully expect a myrinet
10gbase-t nic to cost at least around $500. that sounds like a lot more
than a memfree IB nic, but if IB switch ports cost more than 10gbase-t
ports, it may be irrelevant. besides, I'd actually prefer to pay a bit
more for my nics - to put intelligence in the nic rather than the
I can not comment on the switch side, but I do believe the switch prices
become more attractive.
More information about the Beowulf