FastE NIC Recommendation

[Steffen Persvold]

On Mon, 18 Nov 2002, Donald Becker wrote:

> On Mon, 18 Nov 2002, Steffen Persvold wrote:
> > On Mon, 18 Nov 2002, Trent Piepho wrote:
> > > On Mon, 18 Nov 2002, Steffen Persvold wrote:
> > > > The Intel e1000 (which is in the Linux kernel tree btw.) has a ping-pong/2 
> > > > latency of about 25us on a decent Platform (Xeon, E7500 chipset). The 
> > > > Broadcom 5700 series (with the tg3 driver) has approx. 30us ping-pong/2 
> > > > latency on a ServerWorks GC-LE chipset (also Xeon processors). _But_ if 
> > > > you try e1000 on a AMD 760 MPX chipset for instance, the latency increases 
> > > > to 65 us ping-pong/2. YMMV.
> > > 
> > > Is that an interaction between the 760MPX and the e1000, or typical for the
> > > 760MPX?  ie. does the broadcom 5700 also have high latency with AMD chipsets?
> >
> > Unfortunately I haven't tested the Broadcom adapters on 760 MPX yet 
> > (actually the only reason I've tested Broadcom at all was that it was 
> > onboard on our Dell 2650s). However, I've tested e1000 on PentiumIII 
> > w/ServerWorks HE-SL chipset, and there the latency was 30us. Maybe 
> > there's something with the AMD IOAPICs ? From what I've seen it seem like 
> > most of the latency is produced in the receiving node IRQ processing.
> 
> Yes, that's where a significant part of the increased latency is.
> Compare this _difference_ to the quoted very low message latency of
> cluster network adapters -- the numbers don't make sense.
> 
> The reason is they are comparing different latencies.  Those adapters
> are quoting the time when the CPU is doing nothing but polling for the
> arrival of a message, with the cache hot.  The Gigabit Ethernet latency
> is measured when the system is busy doing other work and is interrupted
> to process an unexpected new message.

Yep, I'm not saying Gigabit latencies are bad. It is a quite different 
message strategy compared to SCI for example (where the sender node CPU 
puts the message directly into the receive node buffer, no DMA startup, no 
interrupt = low latency). What I was making a point of, was that the 
latencies were different on the different kinds of chipsets (i.e AMD MPX 
has 2x times the latency of Intel E7500 with the same kernel, same 
ethernet chip, same driver and same parameters).


> 
> Unlike the case of a global shared memory, low latency is not always a
> good thing.  Gigabit Ethernet adapters go to some effort to _increase_
> latency so that the CPU can process multiple messages during each
> interrupt.  This interrupt mitigation might even be clever enough to
> look at the destination of the next incoming packet to decide if the
> interrupt should be deferred.
> 

Right, and at this point the tg3 and e1000 drivers are doing a quite good 
job at it (AFAIK you can use ethtool to control this interrupt behaviour).

Regards,
--
  Steffen Persvold   |       Scali AS      
 mailto:sp at scali.com |  http://www.scali.com
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY