[Beowulf] Low cost IB cards

[Joe Landman]

Michael H. Frese wrote:
> Be sure your OS -- as in "cat /etc/issue" -- has all the bells and 
> whistles for IB.  We bought those 4X SDR cards and an 8 port switch and 
> tried and tried and tried for over a year off and on and finally failed 
> on various flavors of Fedora to bring up OFA/OFED on IB.  We finally 

Last I checked, OFED isn't supported by Fedora.  Understand, Fedora *is* 
a rapidly moving target.

I don't have anything against it, I just used it to replace a failing 
Ubuntu 9.10 load on a home franken-machine.  But I know that lots of 
things won't work on it.  Right now I am struggling with their ideology 
... nouveau versus Nvidia.  I want to remove the former and install the 
latter.  And it ain't easy.

OFED is picky about its kernels.  No matter which distro you use, 
kernels matter.

> succeeded with CentOS 5.3 or 4.  Then we brought up CentOS on a mixed 
> cluster and now its NFS is failing miserably to communicate with some of 
> the older OS's, on which problem you hear more later....

Older RHEL kernels that Centos are based upon aren't terribly good at NFS.

> 
> Ugh!
> 
> There didn't seem to be any hardware problems, though.

We typically build and install our own kernels.  Its hard to stabilize 
RHEL kernels at high data rates from IO systems (and networks, but thats 
another story).

This said, there are lots of crappy hardware bits out there.  IB tends 
to be pretty good.


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615