[Beowulf] How Would You Test Infiniband in New Cluster?

[Martin Siegert]

On Tue, Nov 17, 2009 at 05:01:12PM -0800, Jon Forrest wrote:
> I had said "I believe these are with IB."
> Now I'm not so sure. I just did a
>
> 	"ifconfig ib0"
>
> on all the nodes and they all say
>
>           BROADCAST MULTICAST  MTU:65520  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:256
>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

AFAIK, ifconfig ib0 will show you the ipoib numbers. Since MPI
(hopefully) is not using this, you see zeros.

> So, it doesn't look like any of these tests used IB,
> although I'm not sure because some of those numbers
> looked better than I'd expect for just 1Gb ethernet.
>
> I'll have to figure out how to force IB when
> using OpenMPI.

Edit your ~/.openmpi/mca-params.conf file and add the line

btl = ^tcp

That will explicitly prevent openmpi using tcp (it would use ib before
tcp by default, but this way it will fail if ib is not working).

> -- 
> Jon Forrest
> Research Computing Support
> College of Chemistry
> 173 Tan Hall
> University of California Berkeley
> Berkeley, CA
> 94720-1460
> 510-643-1032
> jlforrest at berkeley.edu
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

Cheers,
Martin

-- 
Martin Siegert
Head, Research Computing
WestGrid Site Lead
IT Services                                phone: 778 782-4691
Simon Fraser University                    fax:   778 782-4242
Burnaby, British Columbia                  email: siegert at sfu.ca
Canada  V5A 1S6