[Beowulf] Re: typical latencies for gigabit ethernet

[Dave Love]

Patrick Geoffray <patrick at myri.com> writes:

> So, if you set rx-frames to 1, there will be an interrupt after each 
> packet.

Isn't that turning off coalescence, as you recommended?

> Not many devices implement rx-frames, since it does not 
> distinguish between small and large frames. Adaptive coalescing methods 
> do look at the size of the frames to figure out if the traffic is mostly 
> latency or bandwidth sensitive, but it's just a guess.

Yes.  With e1000, I saw 28.2μs omx_perf latency using
InterruptThrottleRate=1, v. 19.5 using InterruptThrottleRate=0.  With
forcedeth, optimization_mode=1, it was 20.2, v. 10.4 with
optimization_mode=0 (the default).  These weren't with the same setup
as the figures on the web page I referred to, by the way.

> On GigE, each 1500 Bytes frames takes more than 10us on the wire so even 
> with interrupt coalescing turned off, you won't get more than 100K 
> interrupts per second.

Good point.

> In the worst case, you would lose a core if you don't let the OS move
> the interrupt handler to do load balancing. What is one core these
> days ? :-)

I guess that depends on what everything else is doing.  It's normally
recommended to use the default (non-)affinity of interrupts, isn't it?

I'll try to collect anything useful from this for the Open-MX FAQ.  This
stuff seems generally badly documented (such as ethtool not even telling
you what the coalescence parameters actually are).  Thanks (and thanks
to Myricom for funding Open-MX, by the way).