[Beowulf] 1.2 us IB latency?

[Peter Kjellstrom]

On Wednesday 28 March 2007, Mark Hahn wrote:
> >> a dramatic breakthrough: 1.2 us, 25M msg/s.  since we just happend to
> >>
> >> or is this an example of message aggregation?  heck, from the url
> >> above, it might even be counting intra-box messages.
> >
> > Nope, this is "normal" ping-pong for the new generation cards (connectx).
>
> so if I do this:
>
> start timer
> send(other,small-message)		recv(first,small-message)
> recv(other,small-message)		send(first,small-message)
> stop timer
>
> I'll actually see 2.4 us between the timer calls?  if I understand,
> aggregation would only help on a streaming test.  in fact, this kind
> of isolated RPC-like exchange is what I see most commonly.

Assuming you could time it with any accuracy, yes. I've seen ib_write_lat 
figures of ~1.

> > Maybe a bit optimistic though, I'd expect closer to 1.5 in a back-to-back
> > config.
>
> so for a small switch (24pt, say), how many hops to the internal fabric,
> and they're, what, .2 us each?

Current switch chips are 24-port, so that means one hop (and yes, I think they 
are around .2 us each).

> also, does back-to-back work well?  I can imagine some cases where
> putting two dual-port cards in each node and creating a mesh might
> work well.

That would require IB-to-IB routing on the hosts, I havn't heard of anyone 
doing that (don't think it's even implemented today).

/Peter

> >> also, I'm sorta amazed people keep selling (and presumably buying)
> >> dual-port IB cards.  doesn't that get quite expensive, switch-wise?
> >
> > Not defending them but, It could possibly maybe be useful if you have a
> > stand-alone IB net for, say, storage or something else not mpi. Also,
> > it's not like they're that much more expensive than single port ones...
>
> yeah, I can see PHB's buying redundant fabrics.  I'd be more interested in
> using the higher port-count for FNN or related topologies (assuming
> switches are cheap, at least at some size...)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://www.scyld.com/pipermail/beowulf/attachments/20070328/6dbc2721/attachment.bin