[Beowulf] Home beowulf - NIC latencies

[Craig Tierney]

On Fri, 2005-02-04 at 13:20, Andrew Piskorski wrote:
> On Thu, Feb 03, 2005 at 04:53:27AM +0100, Vincent Diepeveen wrote:
> 
> > Please note MPI is probably what i'll use, though i keep finding
> > online information about 'gamma'. Is that faster latency than MPI
> > implementations?
> 
>   http://www.disi.unige.it/project/gamma/
> 
> Gamma is a non-TCP/IP Linux 2.6.x network driver for Intel Pro/1000
> gigabit ethernet cards, for use with MPI.  It offers much better
> latency (11 us or so) than TCP/IP over ethernet (maybe 60 or 100 us),
> but worse than the specialized HPC interconnects (maybe 3 us).

See Josip's post on tweaking interrupts on gigE drivers, but
I have a small system with Intel gigE cards and a Dell gigE switch.
Latency between two nodes through the swtich is 30 us.  This is
typical of what I see for other gigE cards.  A latency of 60-100
is a bit high.  

Avoiding TCP/IP is still a big improvement.

Craig

> 
> The attraction of GAMMA, is that Intel Pro/1000 cards can be had for
> $11 to $60 or so each (depending on exact model, etc.), and gigabit
> switches are also pretty cheap, while SCI or Myrinet is somewhere in
> the $500 to $1500 per node range (I don't keep track).
> 
> So if your application can benefit from lower latency, but you want
> something really cheap, GAMMA should be well worth trying.