[Beowulf] 1.2 us IB latency?

[Mark Hahn]

>> start timer
>> send(other,small-message)		recv(first,small-message)
>> recv(other,small-message)		send(first,small-message)
>> stop timer
>>
>> I'll actually see 2.4 us between the timer calls?  if I understand,
>> aggregation would only help on a streaming test.  in fact, this kind
>> of isolated RPC-like exchange is what I see most commonly.
>
> Assuming you could time it with any accuracy, yes.

that's not an issue - rdtsc is perfectly good into the tens of ns range.

> I've seen ib_write_lat figures of ~1.

well, I was assuming mpi - does anyone really write apps using ib primitives?

in anycase, if this is a non-streaming latency result, it's pretty good;
enough to make quadrics look comprehensively out of the picture (guess
they've switched horses to 10GE anyway.)

> That would require IB-to-IB routing on the hosts, I havn't heard of anyone
> doing that (don't think it's even implemented today).

I guess that's part of the reason I'm pulling for 10GE to beat IB:
everything will "just work" as expected.  tcpdump, routing, etc.
even so, IB having reliable multicast and potentially smarter routing
is some advantage, just not sure it's enough to make up for the fully
incompatible infrastructure and somewhat hostile media.