[Beowulf] Gigabit Ethernet and RDMA

[Scott Atchley]

Hi Gus,

Are you trying to find software for NICs you currently have? Or are  
you looking for gigabit Ethernet NICs that natively support some form  
of kernel-bypass/zero-copy?

I do not know of any of the latter (do Chelsio or others offer 1G NICs  
with iWarp?).

As for the former, there are several options for cluster use:

I believe Scyld has an optimized Ethernet stack. GAMMA has special  
drivers for certain Intel NICs. PM/Ethernet-HXB is under active  
development. Open-MX works over any Ethernet driver and works with any  
MPI that works with native MX.

If you are interested in 10G Ethernet, MX on Myricom 10G NICs can work  
with our Myrinet switches or any brand of Ethernet switches (some  
Ethernet switches provide lower latency than others).

Scott

On Aug 11, 2008, at 4:28 PM, Gus Correa wrote:

> Hello Beowulf fans
>
> Does anyone know the status of  RDMA on Gigabit Ethernet?
>
> Is it a stable solution for a cluster interconnect, or still an  
> experimental thing?
>
> Is it effective in offloading network tasks from the CPU?
> (Myrinet and Infiniband seem to use RDMA effectively, right?)
>
> What does it take for it to work under typical Linux distributions?  
> A driver?
> A special kernel?
> Something else? Just plug the NIC in and play?
>
> Does it support standard MPICH2 and/or OpenMPI compiled out of the  
> box,
> or does it require linking to some type of special low level  
> communication library,
> or perhaps requires the use of a special flavor or MPI (say, from  
> the NIC vendor)?
>
> I poked around on the web,
> and learned that Ammasso seems to have pioneered RDMA-enabled GigE  
> NICs (Ammasso 1100).
> Broadcom advertises a NIC with similar characteristics (BCM5706).
> However, it is unclear if RDMA GigE NICs would work with standard  
> Linux distros,
> if it is effective, how much it costs, and how much hassle is  
> required to make it work.
>
> Thank you,
> Gus Correa
>
> -- 
> ---------------------------------------------------------------------
> Gustavo J. Ponce Correa, PhD - Email: gus at ldeo.columbia.edu
> Lamont-Doherty Earth Observatory - Columbia University
> P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
> ---------------------------------------------------------------------
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf