[Beowulf] How Would You Test Infiniband in New Cluster?

[Bill Broadley]

Jon Forrest wrote:
> I had said "I believe these are with IB."
> Now I'm not so sure. I just did a

The performance numbers you showed from relay and mpi_nxnlatbw are definitely
much faster than GigE.  Unless it's multiple copies running on a single
machine (thus printing the hostname).

Assuming that it was actually using the interconnect (not multiple copies
running on a single machine)

> 
>     "ifconfig ib0"

I suspect this is for TCPIP over ib, and doesn't show MPI traffic.  You didn't
mention which controllers do you have?  I suspect that there is a tool to show
the various counters on the HCA, let alone on the switch side.

> I'll have to figure out how to force IB when
> using OpenMPI.

Looks like IB to me, might want to do the reverse to see the real
differential, I find it very handy for cost justifying IB on future clusters
based on real application performance.