[Beowulf] Re: typical latencies for gigabit ethernet

[Dave Love]

Scott Atchley <atchley at myri.com> writes:

> When I test Open-MX, I turn interrupt coalescing off. I run  
> omx_pingpong to determine the lowest latency (LL). If the NIC's driver  
> allows one to specify the interrupt value, I set it to LL-1.

Right, and that's what I did before, with sensible results I thought.
Repeating it now on Centos 5.2 and OpenSuSE 10.3, it doesn't behave
sensibly, and I don't know what's different from the previous SuSE
results apart, probably, from the minor kernel version.  If I set
rx-frames=0, I see this:

rx-usec    latency (µs)
20         34.6
12         26.3
6          20.0
1          14.8

whereas if I just set rx-frames=1, I get 14.7 µs, roughly independently
of rx-usec.  (Those figures are probably ±∼0.2µs.)

> If the  
> driver does not allow specifying the actual rate (i.e. it only has  
> predetermined values), then I leave it off.

Right.  (Adaptive coalescence gave significantly higher latency with our
nVidia and Intel NICs.)

For others interested, this affects TCP results similarly to open-mx,
though the base TCP latency is substantially worse, of course.

I was going to write this up for the OMX FAQ, but was loath to without
understanding the tg3 situation.

> The downside is lower throughput for large messages on 10G Ethernet. I  
> don't think it matters on gigabit.

It doesn't affect the ping-pong throughput significantly, but I don't
know if it has any effect on the system overall (other cores servicing
the interrupts) on `typical' jobs.

> Brice and Nathalie have a paper which implements an adaptive interrupt  
> coalescing so that you do not have to manually tune anything:

Isn't that only relevant if you control the firmware?  I previously
didn't really care about free firmware for devices in the same way as
free software generally, but am beginning to see reasons to care.