[Beowulf] Home beowulf - NIC latencies

[Josip Loncaric]

Vincent Diepeveen wrote:
> At 00:29 4-2-2005 -0800, Bill Broadley wrote:
>>
>>Do you know that gigabit is too high latency?

Gigabit Ethernet adapters often need tweaking to deliver reasonable 
latency, bandwidth, and CPU utilization.

For example, if your system uses the e1000 driver (Intel's gigabit 
Ethernet), the default setting is "dynamic Interrupt Throttle Rate" -- 
which means that the card will delay interrupting the CPU by up to about 
130 microseconds after receiving a packet.  Moreover, the "dynamic" part 
causes the network chip microcode to vary this delay in multiples of 
about 16 microseconds, so that different packets will generally 
experience different receive delays.

For the e1000 driver, 
https://lists.dulug.duke.edu/pipermail/dulug/2004-August/015415.html 
recommends using "options e1000 InterruptThrottleRate=80000" (add this 
line to /etc/modules.conf).  Users of this driver may also want to check 
Intel's parameters for e1000 listed at 
http://www.intel.com/support/network/sb/cs-009209.htm#parameters -- just 
don't assume that the default values are appropriate for cluster use.

Other gigabit Ethernet adapters have similar interrupt mitigation 
strategies, all designed to gracefully cope with high packet rates at 
high network speeds.  For cluster use, adjustments are usually advisable.

The basic Rx interrupt mitigation scheme is this: the receiver's CPU 
won't be interrupted until at least N packets have arrived or M 
microseconds have elapsed (whichever comes first).  This clearly adds up 
to M microseconds to network latency.  BTW, one often sees N=6 
(otherwise NFS performance can seriously degrade) and M>=16.  Other 
variants of this basic scheme are possible; but they all mean increased 
latencies.

Finally, don't forget the Tx side interrupt mitigation, or else the 
sending CPU might not be told promptly that it's OK to send more.  The 
default Tx settings are probably fine for full size packets, but if your 
applications send lots of small packets, tweaking your network driver's 
Tx settings may help.

Sincerely,
Josip