We have the same switch.<br>I was able to get it to work with the latest OFED 1.3 ( available from the Mellanox web site).<br>They have rpms for RHEL4 and RHEL5.<br><br>Massimiliano<br><br><div class="gmail_quote">On Wed, Jun 4, 2008 at 3:52 PM, Michael H. Frese <<a href="mailto:Michael.Frese@numerex-llc.com">Michael.Frese@numerex-llc.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<font size="3">Following Jeff Layton's post to this list [Cheap SDR IB] on
January 28,<br>
we purchased 8 Infinihost LX's and an 8 port switch, and began trying to
get<br>
the OpenFabrics (OFED) release of MVAPICH for Fedora Core 6 to run on our
new<br>
machines. We develop and run a multiphysics code in a relatively
fine grain parallel mode<br>
where latency dominates the performance scaling, so it seemed like a good
thing to try.<br><br>
This is our first exposure to InfiniBand, though we have
considerable<br>
experience with MPI, both in-memory and over GigE, including using
netpipe to<br>
measure latency and bandwidth.<br><br>
Those machines have AMD Athlon X2 6000+'s on Asus M2N-SLI Deluxe
motherboards<br>
with an open PCI Express slot that will handle x4.<br><br>
The main issue is that we are presently running Fedora Core 8 and the
2.6.21<br>
SMP kernel, but there is no OFED release for FC8 yet. Is anyone
else working<br>
on this? Has anyone succeeded at getting it to work?<br><br>
We started with OFED version 1.2.5 from <br>
<a href="http://www.openfabrics.org/downloads/OFED/ofed-1.2.5/OFED-1.2.5-RPMS/" target="_blank">
http://www.openfabrics.org/downloads/OFED/ofed-1.2.5/OFED-1.2.5-RPMS/<br>
</a>We downloaded all the rpms from redhat-release-4AS-6.1 version.<br>
In particular the kernel rpms are kernel-ib-devel-1.2-2.6.9_55.ELsmp
and<br>
kernel-ib-1.2-2.6.9_55.ELsmp.<br><br>
We used the 1.2.5 version because there don't seem to be any rpms for the
1.3 version. <br><br>
All the OFED rpm's for FC6 installed on FC8 without difficulty, except
for opensm-3.0.3-0.ppc64.rpm<br>
It didn't say "missing dependencies ..." It just got
stuck. We had to kill the 'rpm -ivh', remove the lock file <br>
and rebuild the rpm database. After that,<br><br>
# lsmod | grep ib<br><br>
shows about 15 IB related kernel mods.<br><br>
Even so, at this point, some of the IB stuff works. We can run
ibnetdiscover and see the HCA's on the<br>
two machines that have the rpm's installed, and the switch, too. We
could use<br>
that to make a topology file, but we don't know where to put it, or even
if we<br>
should put it somewhere. We can run ibchecknet, and though it finds 4
nodes,<br>
it says they are all bad. It also reports "lid 0 address
resolution:<br>
FAILED". We have not succeeded in getting ibping to work, and
aren't really<br>
sure what how to specify the remote address for it.<br><br>
We found <br><br>
/usr/share/doc/ofed-docs-1.2/README.txt<br>
/usr/share/doc/ofed-docs-1.2/OFED_Installation_Guide.txt<br><br>
and, as described there, did<br><br>
# /etc/init.d/openibd
start
<br>
Loading QLogic InfiniPath
driver:
[FAILED]<br>
Loading HCA driver and Access
Layer:
[ OK ]<br>
Setting up InfiniBand network interfaces:<br>
Failed to configure IPoIB connected mode for ib0<br>
Bringing up interface
ib0:
[FAILED]<br>
Setting up service network . .
.
[ done ]<br>
Loading
ib_sdp
[FAILED]<br>
Loading
ib_vnic
[FAILED]<br>
Module ib_vnic not loaded.<br>
Bringing up VNIC
interfaces
[FAILED]<br><br>
That mostly looks bad.<br><br>
Does anyone have any suggestions?<br><br>
We are willing to try a build from source, but we are unsure of what
challenges might lie down that path.<br><br>
We'd rather not fall back to FC6, but we may have to do that.<br><br>
Thanks for your help.<br><br>
<br>
Mike Frese <br>
</font></div>
<br>_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a><br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
<br></blockquote></div><br>