[Beowulf] Home beowulf - NIC latencies

[Vincent Diepeveen]

Thanks for your deep inside, this is very helpful!

Vincent
www.diep3d.com

At 12:57 4-2-2005 -0700, Josip Loncaric wrote:
>Vincent Diepeveen wrote:
>> At 00:29 4-2-2005 -0800, Bill Broadley wrote:
>>>
>>>Do you know that gigabit is too high latency?
>
>Gigabit Ethernet adapters often need tweaking to deliver reasonable 
>latency, bandwidth, and CPU utilization.
>
>For example, if your system uses the e1000 driver (Intel's gigabit 
>Ethernet), the default setting is "dynamic Interrupt Throttle Rate" -- 
>which means that the card will delay interrupting the CPU by up to about 
>130 microseconds after receiving a packet.  Moreover, the "dynamic" part 
>causes the network chip microcode to vary this delay in multiples of 
>about 16 microseconds, so that different packets will generally 
>experience different receive delays.
>
>For the e1000 driver, 
>https://lists.dulug.duke.edu/pipermail/dulug/2004-August/015415.html 
>recommends using "options e1000 InterruptThrottleRate=80000" (add this 
>line to /etc/modules.conf).  Users of this driver may also want to check 
>Intel's parameters for e1000 listed at 
>http://www.intel.com/support/network/sb/cs-009209.htm#parameters -- just 
>don't assume that the default values are appropriate for cluster use.
>
>Other gigabit Ethernet adapters have similar interrupt mitigation 
>strategies, all designed to gracefully cope with high packet rates at 
>high network speeds.  For cluster use, adjustments are usually advisable.
>
>The basic Rx interrupt mitigation scheme is this: the receiver's CPU 
>won't be interrupted until at least N packets have arrived or M 
>microseconds have elapsed (whichever comes first).  This clearly adds up 
>to M microseconds to network latency.  BTW, one often sees N=6 
>(otherwise NFS performance can seriously degrade) and M>=16.  Other 
>variants of this basic scheme are possible; but they all mean increased 
>latencies.
>
>Finally, don't forget the Tx side interrupt mitigation, or else the 
>sending CPU might not be told promptly that it's OK to send more.  The 
>default Tx settings are probably fine for full size packets, but if your 
>applications send lots of small packets, tweaking your network driver's 
>Tx settings may help.
>
>Sincerely,
>Josip
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
>
>