[Beowulf] How Would You Test Infiniband in New Cluster?

[Gus Correa]

Martin Siegert wrote:
> On Tue, Nov 17, 2009 at 05:01:12PM -0800, Jon Forrest wrote:
>> I had said "I believe these are with IB."
>> Now I'm not so sure. I just did a
>>
>> 	"ifconfig ib0"
>>
>> on all the nodes and they all say
>>
>>           BROADCAST MULTICAST  MTU:65520  Metric:1
>>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>           collisions:0 txqueuelen:256
>>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
> 
> AFAIK, ifconfig ib0 will show you the ipoib numbers. Since MPI
> (hopefully) is not using this, you see zeros.
> 
>> So, it doesn't look like any of these tests used IB,
>> although I'm not sure because some of those numbers
>> looked better than I'd expect for just 1Gb ethernet.
>>
>> I'll have to figure out how to force IB when
>> using OpenMPI.
> 
> Edit your ~/.openmpi/mca-params.conf file and add the line
> 
> btl = ^tcp
> 
> That will explicitly prevent openmpi using tcp (it would use ib before
> tcp by default, but this way it will fail if ib is not working).
> 

Hi Jon

Martin's suggestion is the the best, particularly if you plan to
always use IB, never use TCP.

Alternatively you could
include these mca parameters on the mpiexec command
line to select IB:

-mca btl openib,sm,self

OpenMPI has several mechanisms to make these choices.
See these FAQ:
http://www.open-mpi.org/faq/?category=sysadmin#sysadmin-mca-params
http://www.open-mpi.org/faq/?category=tuning#setting-mca-params

My $0.02
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

>> -- 
>> Jon Forrest
>> Research Computing Support
>> College of Chemistry
>> 173 Tan Hall
>> University of California Berkeley
>> Berkeley, CA
>> 94720-1460
>> 510-643-1032
>> jlforrest at berkeley.edu
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 
> Cheers,
> Martin
>