low-latency high-bandwidth OS bypass user-level messaging for commodity(linux) clusters with commodity NICs(<$200), HELP! (GAMMA/EMP/M-VIA/etc.)

[jon]

> -----Original Message-----
> From: Donald Becker [mailto:becker at scyld.com]
> Sent: Monday, December 16, 2002 6:32 PM
> To: jon
> Cc: beowulf at beowulf.org
> Subject: Re: low-latency high-bandwidth OS bypass user-level messaging
for
> commodity(linux) clusters with commodity NICs(<$200), HELP!
(GAMMA/EMP/M-VIA/etc.)
> 
> On Mon, 16 Dec 2002, jon wrote:
> 
> > Perhaps this isn't the best way to get ahold of you, but I've also
sent
> > this to the Beowulf list.  I've noted your comments on OS Bypass
drivers
> > in the past.  But isn't there some room for non-TCP/IP related
traffic,
> > such as with computing clusters?  We don't need no stinking TCP!  No
> > associated revenue?  You could replace Myrinet in the thousands of
nodes
> > we have here ALONE at NCSA.
> 
> Very likely not.  Myrinet has both a cost and increasing performance
> advantage over gigabit Ethernet when the switch is larger than about
96
> ports.

[jon] I see.

> 
> > We (UIUC theoretical astrophysics group) are in the midst of
purchasing
> > a $50K cluster (I know, small, but big for us! :)) and I'm done all
the
> > research as to what we should be getting.  We ended up going with a
> > Intel Desktop gigabit board and P4, but have found the tests to be
very
> > poor.  We only have 4 nodes right now because we worried about this
very
> > thing.
> 
> Latency or bandwidth?  And what are you using to test?

[jon] We need (projected 0 message size) latencies to be about 30-40us
for our applications.  We end up with message sizes ranging from
256bytes to 8KB for different applications.  A 2.4Ghz P4 cluster has an
idling CPU due to the latency.

> While there are better Gigabit chips than the DP83820, most of its bad
> reputation comes from the poor performance of the other drivers out
> there.  We get quite reasonable performance from it with the Scyld
> ns820.c driver.  Others have reported a 2.5-3X performance improvement
> over the driver written by Red Hat.

[jon] I see, what kinda of 0message latency and peak bandwidth do you
get on 64-bit 66Mhz bus?

[jon] Thanks! -Jon