[Beowulf] Re: typical latencies for gigabit ethernet

[Scott Atchley]

On Jun 29, 2009, at 5:59 AM, Dave Love wrote:

>> Can you say something about any tuning you did to get decent results?
>
> To get the lowest latency, turn off rx interrupt coalescence, either
> with ethtool or module parameters, depending on the driver.  Of  
> course,
> you may not want to turn it off completely, depending on how much load
> the extra interrupts cause.

When I test Open-MX, I turn interrupt coalescing off. I run  
omx_pingpong to determine the lowest latency (LL). If the NIC's driver  
allows one to specify the interrupt value, I set it to LL-1. If the  
driver does not allow specifying the actual rate (i.e. it only has  
predetermined values), then I leave it off.

The downside is lower throughput for large messages on 10G Ethernet. I  
don't think it matters on gigabit.

Brice and Nathalie have a paper which implements an adaptive interrupt  
coalescing so that you do not have to manually tune anything:

"Finding a Tradeoff between Host Interrupt Load and MPI Latency over  
Ethernet" available at:

http://hal.inria.fr/inria-00397328

I do not know if it has been included in a release yet.

Scott