[Beowulf] Home beowulf - NIC latencies

[duncan.roweth at quadrics.com]

Patrick, Vincent

Some input into your discussion. Here is the data
on get latency for Elan4 in an Opteron cluster

quorumi: prun -N2 pgping -f get 0 64
  1:        4 bytes      2.36 uSec     1.69 MB/s
  1:        8 bytes      2.35 uSec     3.40 MB/s
  1:       16 bytes      2.38 uSec     6.73 MB/s
  1:       32 bytes      2.37 uSec    13.50 MB/s
  1:       64 bytes      2.43 uSec    26.30 MB/s

This example reports the average time for 1000
blocking get calls. Patrick's description of the
mechanism is essentially correct, apart from the
detail that we have a fast path for short operations
that avoids the need to set up a DMA. 

You can probably do a bit better on the very fastest
nodes, but this is what I see on the system we have 
in the office. 

> You want less than 3us per Get of 64 Bytes ? I don't 
> know if even Quadrics can do it. 

Yes we can!

> The good news is that you can pipeline it very well. 

Indeed. There is lots of parallelism in the hardware
so you can me processing multiple requests at the same
time. In this sequence of short jobs I measure the 
average time for 8 byte gets 2 at a time, 4 at a time
etc. 

quorumi: prun -N2 pgping -f get -b2 8
  1:        8 bytes      1.32 uSec     6.07 MB/s
quorumi: prun -N2 pgping -f get -b4 8
  1:        8 bytes      1.04 uSec     7.66 MB/s
quorumi: prun -N2 pgping -f get -b8 8
  1:        8 bytes      0.84 uSec     9.47 MB/s
quorumi: prun -N2 pgping -f get -b16 8
  1:        8 bytes      0.82 uSec     9.79 MB/s
quorumi: prun -N2 pgping -f get -b32 8
  1:        8 bytes      0.79 uSec    10.18 MB/s

The limiting factor is the rate at which the remote NIC
can read data over the PCI bus. 

Best Wishes
Duncan Roweth
Quadrics Limited


P.S. Clearly our sales people focus on the current product 
(Elan4 NICs) but we will be supporting the installed base
of Elan3 systems for some years yet. Most of the big systems
have extended warranties, so we keep a stock of spares, but 
there are a few hundred adapters and associated switches. 
Drop us some mail if you are interested.