[Beowulf] Re: typical latencies for gigabit ethernet
d.love at liverpool.ac.uk
Tue Jun 30 06:17:35 PDT 2009
Patrick Geoffray <patrick at myri.com> writes:
> So, if you set rx-frames to 1, there will be an interrupt after each
Isn't that turning off coalescence, as you recommended?
> Not many devices implement rx-frames, since it does not
> distinguish between small and large frames. Adaptive coalescing methods
> do look at the size of the frames to figure out if the traffic is mostly
> latency or bandwidth sensitive, but it's just a guess.
Yes. With e1000, I saw 28.2μs omx_perf latency using
InterruptThrottleRate=1, v. 19.5 using InterruptThrottleRate=0. With
forcedeth, optimization_mode=1, it was 20.2, v. 10.4 with
optimization_mode=0 (the default). These weren't with the same setup
as the figures on the web page I referred to, by the way.
> On GigE, each 1500 Bytes frames takes more than 10us on the wire so even
> with interrupt coalescing turned off, you won't get more than 100K
> interrupts per second.
> In the worst case, you would lose a core if you don't let the OS move
> the interrupt handler to do load balancing. What is one core these
> days ? :-)
I guess that depends on what everything else is doing. It's normally
recommended to use the default (non-)affinity of interrupts, isn't it?
I'll try to collect anything useful from this for the Open-MX FAQ. This
stuff seems generally badly documented (such as ethtool not even telling
you what the coalescence parameters actually are). Thanks (and thanks
to Myricom for funding Open-MX, by the way).
More information about the Beowulf