[Beowulf] Home beowulf - NIC latencies
josip at lanl.gov
Fri Feb 4 11:57:28 PST 2005
Vincent Diepeveen wrote:
> At 00:29 4-2-2005 -0800, Bill Broadley wrote:
>>Do you know that gigabit is too high latency?
Gigabit Ethernet adapters often need tweaking to deliver reasonable
latency, bandwidth, and CPU utilization.
For example, if your system uses the e1000 driver (Intel's gigabit
Ethernet), the default setting is "dynamic Interrupt Throttle Rate" --
which means that the card will delay interrupting the CPU by up to about
130 microseconds after receiving a packet. Moreover, the "dynamic" part
causes the network chip microcode to vary this delay in multiples of
about 16 microseconds, so that different packets will generally
experience different receive delays.
For the e1000 driver,
recommends using "options e1000 InterruptThrottleRate=80000" (add this
line to /etc/modules.conf). Users of this driver may also want to check
Intel's parameters for e1000 listed at
http://www.intel.com/support/network/sb/cs-009209.htm#parameters -- just
don't assume that the default values are appropriate for cluster use.
Other gigabit Ethernet adapters have similar interrupt mitigation
strategies, all designed to gracefully cope with high packet rates at
high network speeds. For cluster use, adjustments are usually advisable.
The basic Rx interrupt mitigation scheme is this: the receiver's CPU
won't be interrupted until at least N packets have arrived or M
microseconds have elapsed (whichever comes first). This clearly adds up
to M microseconds to network latency. BTW, one often sees N=6
(otherwise NFS performance can seriously degrade) and M>=16. Other
variants of this basic scheme are possible; but they all mean increased
Finally, don't forget the Tx side interrupt mitigation, or else the
sending CPU might not be told promptly that it's OK to send more. The
default Tx settings are probably fine for full size packets, but if your
applications send lots of small packets, tweaking your network driver's
Tx settings may help.
More information about the Beowulf