[Beowulf] Re: typical latencies for gigabit ethernet
d.love at liverpool.ac.uk
Mon Jun 29 09:10:03 PDT 2009
Scott Atchley <atchley at myri.com> writes:
> When I test Open-MX, I turn interrupt coalescing off. I run
> omx_pingpong to determine the lowest latency (LL). If the NIC's driver
> allows one to specify the interrupt value, I set it to LL-1.
Right, and that's what I did before, with sensible results I thought.
Repeating it now on Centos 5.2 and OpenSuSE 10.3, it doesn't behave
sensibly, and I don't know what's different from the previous SuSE
results apart, probably, from the minor kernel version. If I set
rx-frames=0, I see this:
rx-usec latency (µs)
whereas if I just set rx-frames=1, I get 14.7 µs, roughly independently
of rx-usec. (Those figures are probably ±∼0.2µs.)
> If the
> driver does not allow specifying the actual rate (i.e. it only has
> predetermined values), then I leave it off.
Right. (Adaptive coalescence gave significantly higher latency with our
nVidia and Intel NICs.)
For others interested, this affects TCP results similarly to open-mx,
though the base TCP latency is substantially worse, of course.
I was going to write this up for the OMX FAQ, but was loath to without
understanding the tg3 situation.
> The downside is lower throughput for large messages on 10G Ethernet. I
> don't think it matters on gigabit.
It doesn't affect the ping-pong throughput significantly, but I don't
know if it has any effect on the system overall (other cores servicing
the interrupts) on `typical' jobs.
> Brice and Nathalie have a paper which implements an adaptive interrupt
> coalescing so that you do not have to manually tune anything:
Isn't that only relevant if you control the firmware? I previously
didn't really care about free firmware for devices in the same way as
free software generally, but am beginning to see reasons to care.
More information about the Beowulf