[Beowulf] How Would You Test Infiniband in New Cluster?
siegert at sfu.ca
Tue Nov 17 18:30:00 PST 2009
On Tue, Nov 17, 2009 at 05:01:12PM -0800, Jon Forrest wrote:
> I had said "I believe these are with IB."
> Now I'm not so sure. I just did a
> "ifconfig ib0"
> on all the nodes and they all say
> BROADCAST MULTICAST MTU:65520 Metric:1
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:256
> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
AFAIK, ifconfig ib0 will show you the ipoib numbers. Since MPI
(hopefully) is not using this, you see zeros.
> So, it doesn't look like any of these tests used IB,
> although I'm not sure because some of those numbers
> looked better than I'd expect for just 1Gb ethernet.
> I'll have to figure out how to force IB when
> using OpenMPI.
Edit your ~/.openmpi/mca-params.conf file and add the line
btl = ^tcp
That will explicitly prevent openmpi using tcp (it would use ib before
tcp by default, but this way it will fail if ib is not working).
> Jon Forrest
> Research Computing Support
> College of Chemistry
> 173 Tan Hall
> University of California Berkeley
> Berkeley, CA
> jlforrest at berkeley.edu
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
Head, Research Computing
WestGrid Site Lead
IT Services phone: 778 782-4691
Simon Fraser University fax: 778 782-4242
Burnaby, British Columbia email: siegert at sfu.ca
Canada V5A 1S6
More information about the Beowulf