[Beowulf] Home beowulf - NIC latencies

[Douglas Eadline, Cluster World Magazine]

On Fri, 4 Feb 2005, Andrew Piskorski wrote:

> On Thu, Feb 03, 2005 at 04:53:27AM +0100, Vincent Diepeveen wrote:
> 
> > Please note MPI is probably what i'll use, though i keep finding
> > online information about 'gamma'. Is that faster latency than MPI
> > implementations?
> 
>   http://www.disi.unige.it/project/gamma/
> 
> Gamma is a non-TCP/IP Linux 2.6.x network driver for Intel Pro/1000
> gigabit ethernet cards, for use with MPI.  It offers much better
> latency (11 us or so) than TCP/IP over ethernet (maybe 60 or 100 us),
> but worse than the specialized HPC interconnects (maybe 3 us).

The "60-100 us" is incorrect. With proper tuning an e1000 can get 25us 
latency (using netpipe). (see Jossip's post about tunning parameters)
Oh, and by the way this was using a 32 PCI desk top card. 

A low latency number is not the whole story however, processor load
is another issue. The point is that tuning can make a difference. Default
values are usually set for maximum throughput and low CPU overhead.

It all depends on what your application needs. If you need GAMMA,
then that is a good choice, but many applications may work well
with proper tuning of NIC parameters.

As an aside, Netgear used to sell a low cost desktop NIC
(GA302T-tigon3/Broadcom) which had very good numbers as well.
I profiled this NIC in the first issue of ClusterWorld. 


Doug

> 
> The attraction of GAMMA, is that Intel Pro/1000 cards can be had for
> $11 to $60 or so each (depending on exact model, etc.), and gigabit
> switches are also pretty cheap, while SCI or Myrinet is somewhere in
> the $500 to $1500 per node range (I don't keep track).
> 
> So if your application can benefit from lower latency, but you want
> something really cheap, GAMMA should be well worth trying.
> 
> 

-- 
----------------------------------------------------------------
Editor-in-chief                   ClusterWorld Magazine
Desk: 610.865.6061                            
Fax:  610.865.6618                www.clusterworld.com