[Beowulf] Open InfiniBand
Greg M. Kurtzer
gmkurtzer at lbl.gov
Wed Dec 21 10:26:23 PST 2005
I have been experimenting with OpenIB and OpenMPI on several clusters. I
will be building another one today (pending no fires) with cAos2 and
Warewulf. cAos-2 actually has the openIB gen2 stack in the standard
kernel (the 18.104.22.168 in the testing repository). I have seen several
other problems with that kernel on Dual core Opterons systems WRT to PCI
express (which is why that kernel hasn't made it out of the testing
repository yet). Luckily the system I want to run on is a EM64T ;)
cAos-2 also has current MPI's in its ports tree which also integrated
with environment modules. This means you can do:
# cd /usr/src/ports/programs/openmpi/1.0
# make install
And then all users can select the MPI that they want to use by:
$ module load openmpi/1.0
OK, enough evangelizing... Back to the question at hand.
Our initial tests have been promising. Not perfect yet, but good enough
to start testing for production systems. (we would rather utilize a
community maintained solution then a proprietary stack that locks us to
a particular version/infrastructure (obviously)).
On Wed, Dec 21, 2005 at 09:55:55AM +0100, Daniel Pfenniger wrote:
> We have a cluster with InfiniBand network that works flawlessly. The problem is
> that the binary IB drivers are proprietary, and the company has been recently
> absorbed by a larger one, which means that the driver support for Mellanox
> cards is now frozen.
> The proprietary drivers prevents to upgrade or tweak the kernel, or to use
> a recent version of mpich2 (with MPI 2 and one-sided communications).
> Fortunately OpenIB (openib.org) has published open source drivers which are
> included in the latest kernels >= 2.6.14.
> Before exploring this option I just want to ask the list if anyone has used the
> OpenIB drivers for production work, in particular with an open source MPI?
> Thanks, Dan
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Berkeley Lab, Linux guy
More information about the Beowulf