[Beowulf] 1.2 us IB latency?

[Mark Hahn]

>> in anycase, if this is a non-streaming latency result, it's
>> pretty good; enough to make quadrics look comprehensively out
>> of the picture (guess they've switched horses to 10GE anyway.)
>
> It is a non-streaming latency on Mellanox ConnectX. On MPI it
> is 1.2us latency - Pallas MPI Ping.

this is something like 3x better than previous IB - a difference 
of that magnitude must always raise eyebrows.  how was such a 
dramatic improvement made?  what was so badly broken in previous 
incarnations?  is there some caveat to the new score, like requiring
all of userspace to be pre-pinned, or buffers to be 4k aligned or something?

also, just to be perfectly explicit, this is 1.2 us inter-node,
right?  not something crazy like two 8-core boxes with only two
of 16 hops inter-box?

thanks, mark hahn.