Any news on Infiniband ?

[Anthony Skjellum]

Markus,

Because people ask for polling latency (it is not the right model for most
real applications), we did provide this number.  We have found almost universal
advantage to avoid polling MPI in almost all networks, including Ethernet+TCP
Giganet, and Myrinet.    As usual, we did non-polling first, then people
asked to see these sub-ten microsecond numbers to take the IB seriously,
even though their apps will mostly likely be faster in the non-polling mode,
and I believe significantly so (as we have seen before).

Our white paper shows the polling and non-polling implementation curves for
bandwidth.  We normally encourage people not to use polling mode, where
overhead is low.  We have more data on same coming.  So far, the non-polling
latency is a bit higher than giganet was.  30us + right now.  [Because
30us represents a lot of data bit times, we like this to be lower.  We had
25us or so on Giganet a while ago.  Yet, I am confident that HCA/stack
issue is going to get optimized more, from public indications of the
companies we talk to in IB space.]

We continue to update this white paper bi-weekly, as we get new data.
We will do more to clarify overhead for long transfer moving forward than we
have so far.

Here is URL
http://www.mpi-softtech.com/company/publications/files/prog_ib_2003-02-13.pdf

There is no PIO per se in our model.  There is always DMA.  We do work to do
lazy unpinning (like Myrinet does), and we do - as with Giganet - look at the
tradeoff of pinning cost vs. buffering on each side.  Since we are wanting low
overhead for longish messages and overlapping, we do true zero copy for same,
and we do asynchronous progress and we do non-polling completion.

The next release will mention how we go to 8.7us (e.g., you guve multithreaded
MPI support, etc), as we cut back to deliver this familiar metrics.

The best thing - and we are supporting it on early testbeds - is to get real
applications run on IB at scale.

It is important to note that production-level hardware and MPI are just
shipping now/next month, IB has not really been out till March-April time frame
of this year.

Tony

On Wed, 26 Feb 2003, Markus Fischer wrote:

> What about CPU load ?
>
> Is it PIO only or DMA ?
>
> The OSU numbers dropped off right when hitting the L2 cache size ...
>
> Markus
>
> Anthony Skjellum wrote:
>
> >Hi, we have 8.7us for zero byte messages in our MPI/Pro library.  I've
> >heard a rumor that OSU is claiming 7.5 for their experimental MPI.
> >
> >We've seen bandwidths of over 820Mbyte/s with Mellanox gear on page
> >aligned data, with SWGC chipset, but not at 128 byte size transfers.
> >
> >-Tony
> >
> >On Wed, 26 Feb 2003 pascal.deveze at bull.net wrote:
> >
> >
> >
> >>I'd like to get recent information on that new interconnect technology.
> >>
> >>I know that information can be found on different sites :
> >>
> >>http://infiniband.sourceforge.net
> >>http://www.mellanox.com/technology/whitepapers.html
> >>
> >>But, I cannot see recent news and forecast. I'd like to know if
> >>optimisations of hardware and software are planned and what are the
> >>goals in term of :
> >>
> >>- Bandwidth : My last information is that current maximum is about 800 MBytes/s
> >>                    for message size of 128 KBytes
> >>
> >>- Latency (message of 0 0byte) : My last information is that current minimum
> >>                 is about 20 micros.
> >>
> >>- Milestone for optimisations. I do not have any information.
> >>
> >>- Hardware / Software optimizations: I do not have any information
> >>
> >>
> >>_______________________________________________
> >>Beowulf mailing list, Beowulf at beowulf.org
> >>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> >>
> >>
> >>
> >
> >Anthony Skjellum PhD, CTO       | MPI Software Technology, Inc.
> >101 South Lafayette St, Ste. 33 | Starkville, MS 39759, USA
> >Ph: +1-(662)320-4300 x15        | FAX: +1-(662)320-4301
> >
> >http://www.mpi-softtech.com     | tony at mpi-softtech.com
> >
> >Middleware that's hard at work for you and your enterprise.(SM)
> >
> >_______________________________________________
> >Beowulf mailing list, Beowulf at beowulf.org
> >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> >
> >
> >
>
>

Anthony Skjellum PhD, CTO       | MPI Software Technology, Inc.
101 South Lafayette St, Ste. 33 | Starkville, MS 39759, USA
Ph: +1-(662)320-4300 x15        | FAX: +1-(662)320-4301

http://www.mpi-softtech.com     | tony at mpi-softtech.com

Middleware that's hard at work for you and your enterprise.(SM)