[Beowulf] newb question - help with NUMA + mpich2 + GPUs

[Christopher Samuel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 21/02/12 09:52, Micha wrote:

> Things are actually more complex as this is a mixed GPU/GPU (CUDA) 
> system so I'm also looking for how to effectively transfer data
> between GPUs siting on different PCIe slots and find the affinity
> between GPUs and CPUs. Also at what stage is the support for using
> MPI to copy between GPUs?

The hwloc library from the Open-MPI folks will probably help with some
of it:

http://www.open-mpi.org/projects/hwloc/

It can show you which cores are near which PCI devices for instance
and lstopo is a fantastic tool for getting a quick overview of a node.

I *believe* that code is in the 1.5 series but it'd be well worth
asking the question on the open-mpi lists to get a definitive answer
from someone who knows what they're talking about. :-)

There was also a discussion on the Open-MPI devel list recently about
why MVAPICH2 appears to do better than it with GPUs (for the moment),
the summary is here:

http://www.open-mpi.org/community/lists/devel/2012/02/10430.php

Hope this helps!
Chris
- -- 
    Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
         http://www.vlsci.unimelb.edu.au/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9DEAAACgkQO2KABBYQAh8h1gCfRtYtAY6hra6ckeoC60ZkfqOe
qPwAnAsZCHB/5E9QMYutgTMKiW4cdlxO
=cwG5
-----END PGP SIGNATURE-----