[Beowulf] Maximizing intra-node communication performance

[Paulo Afonso Lopes]

Just my 2c:

As Craig said, looking for Shmem is the place to start. I browsed the
LAM-MPI user's guide (I have no experience with it) and they offer 3
implementation options for MPI comms: a) lam_basic, b) smp, c) shmem.

It seems (quick browsing!) that a) and b) are TCP/IP based, with b) tuned
for SMP machines. Not all primitives were implemented in b) and c) (start
reading from page 86), so if you use them, they will use the lam_basic.

Nice machine to play with; not really an SMP, but more of a NUMA. You can
try to get a perfomance baseline by writing a small program that moves
data between 2 threads (threads share memory easily, and you don't need to
write SYSV shmmem code) and i) affining the threads to cores on the same
chip, then ii) moving one of the threads to a core on another chip in the
same board, and then iii) moving it to a core in another board (I browsed
the Tyan site and the 8-way you got is built from 2 4-way "boards"). If
you do them, please report your findings! (BTW, what was the system cost?)

You may also try to "play" with MTU on the localhost interface (OOTH I
can't remember what it the MTU of localhost), but I do not beleive you'll
get something important out of it...

Remember to use a kernel with NUMA extensions!

paulo lopes



> Does LAM-MPI not have a shared memory device for intra-node
> communication?  How about changing your MPI implementation to
> one that does?
>
> Craig
>

>> At 12:05 27-12-2005 +0200, Tahir Malas wrote:
>> >Hi all,
>> >Taking advice from a previous discussion, we have purchased an Tyan
>> server
>> >with 8 dual-core Opteron 870 processors. Now I want to wonder how I can
>> >maximize the intra-node communication of the server. We have been using
>> >LAM-MPI, but I think that TCP/IP protocol may degrade the performance.
>> Has
>> >anybody tried new implementations of MPI, or anybody knows some other
>> >support for intra-node communication?
>> >Thanks in advance,
>> >Tahir Malas
>> >Bilkent University
>> >Electrical and Electronics Engineering Department
>> >Phone: +90 312 290 1385
>> >
>
--
Paulo Afonso Lopes                        | Tel: +351- 21 294 8536
Departamento de Informática               | 294 8300 ext.10763
Faculdade de Ciências e Tecnologia        | Fax: +351- 21 294 8541
Universidade Nova de Lisboa               | e-mail: pal at di.fct.unl.pt
2829-516 Caparica, PORTUGAL