[Beowulf] AMD 8 cores vs 12 cores CPUs and Infiniband
hahn at mcmaster.ca
Tue Mar 29 21:02:33 PDT 2011
> I have been using single card on Magny-Cours with no issues at all. You can
interesting. what adjustments have you made to the MPI stack to permit this?
we've had a variety of apps that fail intermittently on high-core nodes.
I have to say I was surprised such a thing came up - not sure whether it's
inherent to IB or a result of the openmpi stack. our usual way to test
this is to gradually reduce the ranks-per-node for the job until it starts
to work. an interesting cosmology code works at 1 pppn but not 3 ppn
on our recent 12c MC, mellanox QDR cluster.
> Using 16 cores per node (8 * 2) seem the 'safe' option, but the 24 cores
> (12 * 2) is better in term of price per job. Our CFD applications using MPI
> (OpenMPI) may need to do about 15 'MPI_allreduce' calls in one seccond or
> less, and we may probably using a pool of 1500 cores.
but will that allreduce be across 1500 cores? I can get you a scaling curve
for the previously mentioned MC cluster (2.2 GHz).
> 2 - I've heard that QLogic behavior is better in terms of QP creation, I
well, they've often bragged about message rates - I'm not sure how relate
that is to QP creation. I'd be interested to hear of some experiences, too.
regards, mark hahn.
More information about the Beowulf