[Beowulf] 1.2 us IB latency?

[Håkon Bugge]

At 23:37 19.04.2007, greg.lindahl at qlogic.com wrote:
>I'm not sure exactly which results you are referring to, but keep in
>mind that you need to divide the overhead by the number of bytes
>sent. GigE usually sends a lot fewer bytes than InfiniBand.

I ran the benchmark using both IB and Gbe and 
compared... (and I did the division you refer to above)

The reason for the benchmark showing Gbe to have 
less overhead than IB is that the overhead 
associated with Gbe NIC interrupt processing, 
TCP/IP protocol processing, and socket overhead 
is mainly handled by kernel threads running on a 
different core than the MPI process - fooling the 
benchmark to believe Gbe is more efficient than 
IB :-) Lessons learned are to be skeptical to 
benchmarks showing superb result using a single core on a multicore machine ;-)


Håkon