[Beowulf] mpich, mpich2, 64 bit Linux, and 1Gbit network

[Konstantin Kudin]

 Hi there,

 While trying to figure out how an application ( a Car-Parrinello code
from www.pwscf.org) scaled over 1Gbit network I tested different mpich
versions with options.

 The system is a bunch of dual Opterons (246) with the usual 1 Gbit
network and 64-bit CentOS 3, kernel version is 2.4.21-32.ELsmp, and
some kind of NUMA is turned on.

 It turns out that using the shared memory within a node is quite a
challenge with both mpich versions on this OS. MPICH2 (version 1.0.2p1)
simply hangs with [--with-device=ch3:ssm], but works with [=ch3:sock].
On the other hand, MPICH 1.2.7 requires some hoop jumping to work
properly.

 Specifically, P4_GLOBMEMSIZE issue comes up, and to set it to the
value that the code works with (1 Gb or [setenv P4_GLOBMEMSIZE
1073741824]) the code needs to be updated to allow such values in the
file mpid/ch_p4/p4/lib/p4_MD.h [define P4_MAX_SYSV_SHMIDS  1024]. The
default for the latter [ 256 ] is too small.

 Then, a properly compiled mpiexec (version 0.80)  from OSC does not
work with this shared memory mpich-1.2.7, and so mpich again needs to
be updated in file [mpid/ch_p4/p4/OPTIONS] to have uncommented [#define
SYSV_IPC].

 Finally, when the dust settles, the code runs the fastest with only 4
cpus (2nodes by 2cpus), and even 6 cpus are already slower than 4.

 The fastest version is then MPICH1 with shared memory, followed by
MPICH2 with sockets, and the slowest is MPICH1 with sockets.

 Is such scaling normal for the 1Gbit network? Also, could mpich1 have
more reasonable defaults for systems like these?

 Thanks!
 Konstantin

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com