low-latency high-bandwidth OS bypass user-level messaging for commodity(linux) clusters with commodity NICs(<$200), HELP! (GAMMA/EMP/M-VIA/etc.)
jcmcknny at uiuc.edu
Mon Dec 16 22:08:32 PST 2002
What's the deal with SCore?
The guys at GAMMA mentioned them, as well as some friends on mine
recently. It seems to do quite well with Intel gigabit chips (30us
latency and 120MB/sec peak bandwidth).
Why have I never heard of it? How do they achieve this? Not a total
OS-bypass linux distribution? They seem to be an entire linux dist.
Why can't Scyld do this?
> -----Original Message-----
> From: Michael T. Prinkey [mailto:mprinkey at aeolusresearch.com]
> Sent: Monday, December 16, 2002 6:37 PM
> To: jcmcknny at uiuc.edu
> Cc: beowulf at beowulf.org; becker at scyld.com
> Subject: Re: low-latency high-bandwidth OS bypass user-level messaging
> commodity(linux) clusters with commodity NICs(<$200), HELP!
> Hi Jon,
> I began investigating the same issue a few months ago as Cu gigabit
> and switches became widely available. The situation is much the same
> it was then. No one is actively developing bypass support for the new
> networking hardware. The state of MVIA in particular has been in
> at least 18 months. GAMMA is interesting and seems to be more
> developed, but when I last checked there was no support for SMP. The
> web site was promising a second-generation of the core code which
> to make driver development more simple. To my knowledge, that version
> not seen the light of day.
> I think that many people would love to take advantage of newer cheap
> hardware with OS bypass, but as of yet, no one is really taking the
> pulling the drivers together. In my mind, this is the next big hurdle
> the Beowulf community to overcome if we intend to really use commodity
> hardware for network interconnects.
> Mike Prinkey
> Aeolus Research, Inc.
> > Hi Donald Becker, master of all that is networking! And anyone else
> > that can help :)
> > Perhaps this isn't the best way to get ahold of you, but I've also
> > this to the Beowulf list. I've noted your comments on OS Bypass
> > drivers in the past. But isn't there some room for non-TCP/IP
> > traffic, such as with computing clusters? We don't need no stinking
> > TCP! No associated revenue? You could replace Myrinet in the
> > thousands of nodes we have here ALONE at NCSA.
> > We (UIUC theoretical astrophysics group) are in the midst of
> > a $50K cluster (I know, small, but big for us! :)) and I'm done all
> > research as to what we should be getting. We ended up going with a
> > Intel Desktop gigabit board and P4, but have found the tests to be
> > poor. We only have 4 nodes right now because we worried about this
> > very thing.
> > Anyways, our problem is we are willing to pay for a commercial
> > but not at any cost, perhaps upto $200 per board. Basically, we see
> > all these solutions such as:
> > Giganet using VIA
> > ServernetII using VIA
> > InfiniBand
> > U-Net
> > AM II
> > LPC
> > PM
> > FM
> > GigaE-PM
> > BIP
> > EMP
> > GAMMA
> > M-VIA
> > Half of these are seemingly dead, those that seem relatively alive
> > M-VIA: http://www.nersc.gov/research/ftg/via/
> > Only support a few devices, and only 1 expensive ($500) gigabit
> > that's still available (the SysKonnect).
> > GAMMA: http://www.disi.unige.it/project/gamma/index.html
> > Depending on what part of their website you are at, they support
> > different devices. The Alteon TIGON-II results seem to suck for
> > latency, which is our biggest problem. The Netgear GA621 looks
> > But we already bought a $5000 copper gigabit switch! We are stuck
> > it! (HP Procurve 5308xl). Whether they support the GA622 is kinda
> > or at least untested according to the website. No luck getting in
> > touch with driver writer about that. Besides, EMP guy says the
> > sucks!
> > EMP: http://www.osc.edu/~pw/emp/
> > Seems to be interesting, although the available 3Com 3C996, of which
> > have 3 to test, is only said to be "maybe" supported since it's
> > and not Tigon 2. And will it such in latency just like the Tigon 2?
> > EMP guys says the GA-622T sucks with it's ns chipset and that was
> > option with GAMMA, assuming he really did write the driver for both
> > and 622 (their website isn't clear about this, and no emails from
> > guys there), since the 622 "was" an option.
> > Basically after all my testing (about 2 months of light testing and
> > last 2 weeks of hard-core 24-hour a day testing) I realized TCP
> > and I need an OS Bypass or user-level communication driver.
> > My questions are:
> > 1) Is there a commercial product for a not so expensive board that
> > provides what GAMMA/EMP/M-VIA provide? Any other OS-bypass
> > layer I don't know about?
> > 2) Is there a solution I'm missing? Has to be copper gigabit for
> > linux, OS-bypass like GAMMA, MPI on top of that GAMMA-like. No dead
> > boards, etc. Why are there no commercial products? MPI/Pro is just
> > funny MPI still on top of TCP, no?
> > I basically want 20us latency for 0 message size and peak bandwidth,
> > for $100-$200 per board on gigabit. Not too much to ask? :) I
> > it's certainly possible.
> > Currently with Myrinet on P4 I get 17us latency and 80MB/sec
> > gigabit on P4 gets 70us latency and 80MB/sec. On Xeon's I get 50us
> > latency and 95MB/sec bandwidth with latest 3com bcm5700 or latest
> > e1000 driver.
> > I'm going to try EMP with my 3c996, but honestly his instructions
> > damn vague and confusing (i.e. WHAT snapshot of gcc/binutils should
> > use, what the heck do I do?, etc.) Might try GAMMA too since EMP
> > it may work as a Tigon processor. GAMMA seems a bit less crazy.
> > Honestly, I can't really figure out what Scyld does. Is it just a
> > linux distribution? Does it actually have OS-bypass networking?
> > anything?
> > Why is the OS-bypass so hard? If wanting no TCP support, isn't it
> > easier than writing standard linux driver? (like you've done a lot!)
> > Thanks!
> > Jonathan McKinney
> > University of Illinois at Urbana-Champaign
> > Center for Theoretical Astrophysics
> > NCSA
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org
> > To change your subscription (digest mode or unsubscribe) visit
> > http://www.beowulf.org/mailman/listinfo/beowulf
More information about the Beowulf