[Beowulf] MPI programming question: Interleaved MPI_Gatherv?

[Joachim Worringen]

Michael Gauckler wrote:
> Dear List, 
> 
> I would like to gather the data from several processes. 
> Instead of the comonly used stride, I want to interleave 
> the data:
> 
> Rank 0: AAAAA -> ABCDABCDABCDABCDABCD
> Rank 1: BBBBB ----^---^---^---^---^
> Rank 2: CCCCC -----^---^---^---^---^
> Rank 3: DDDDD ------^---^---^---^---^
> 
> Since the stride of the receive type is indicated 
> in multpiles of its mpi_type, no interleaving is 
> possible (the smallest striping factor leads to 
> AAAAABBBBBBCCCCCDDDDD).
> 
> Is there a way to achieve this behaviour in an 
> elegant way, as MPI_Gather promises it? Or do
> I need to do Send/Recv with self-aligned offsets?

Actually, I don't see an 'elegant' way to do this, either. The decision 
between multiple MPI_Gatherv() calls and a Irecv/Send/Waitall construct 
depends on the quality of the MPI implementation you use (MPI_Gatherv 
can be optimized well for small amounts of data), the characteristics of 
you interconnect (high latency gives more room for optimization) and the 
number of processes you use. For small process numbers, you wont see 
much of a difference anyway.

You could also try to gather all data on the root in separate buffers, 
and then let this process send/recv to itself using the proper datatypes.

Finally, if this communication is not a significant part of your 
runtime, you shouldn't spend much time optimizing it anyway.

  Joachim

-- 
Joachim Worringen - NEC C&C research lab St.Augustin
fon +49-2241-9252.20 - fax .99 - http://www.ccrl-nece.de