[Beowulf] MPI Coding help needed

[amjad ali]

Hi all.
(sorry for duplication, if it is)

I have to parallelize a CFD code using domain/grid/mesh partitioning among
the processes. Before running, we do not know,
(i) How many processes we will use ( np  is unknown)
(ii) A process will have how many neighbouring processes (my_nbrs = ?)
(iii) How many entries a process need to send to a particular neighbouring
process.
But when the code run, I calculate all of this info easily.


The problem is to copy a number of entries to an array then send that array
to a destination process. The same sender has to repeat this work to send
data to all of its neighbouring processes. Is this following code fine:

DO i = 1, my_nbrs
   DO j = 1, few_entries_for_this_neighbour
       send_array(j)   =    my_array(jth_particular_entry)
   ENDDO
   CALL MPI_ISEND(send_array(1:j),j, MPI_REAL8, dest(i), tag,
MPI_COMM_WORLD, request1(i), ierr)
ENDDO

And the corresponding receives, at each process:

DO i = 1, my_nbrs
   k = few_entries_from_this_neighbour
   CALL MPI_IRECV(recv_array(1:k),k, MPI_REAL8, source(i), tag,
MPI_COMM_WORLD, request2(i), ierr)
   DO j = 1, few_from_source(i)
       received_data(j)   =    recv_array(j)
   ENDDO
ENDDO

After the above MPI_WAITALL.


I think this code will not work. Both for sending and receiving. For the
non-blocking sends we cannot use send_array to send data to other processes
like above (as we are not sure for the availability of application buffer
for reuse). Am I right?

Similar problem is with recv array; data from multiple processes cannot be
received in the same array like above. Am I right?


Target is to hide communication behind computation. So need non blocking
communication. As we do know value of np or values of my_nbrs for each
process, we cannot decide to create so many arrays. Please suggest solution.


===================
A more subtle solution that I could assume is following:

cc = 0
DO i = 1, my_nbrs
   DO j = 1, few_entries_for_this_neighbour
       send_array(cc+j)   =    my_array(jth_particular_entry)
   ENDDO
   CALL MPI_ISEND(send_array(cc:cc+j),j, MPI_REAL8, dest(i), tag,
MPI_COMM_WORLD, request1(i), ierr)
   cc = cc  + j
ENDDO

And the corresponding receives, at each process:

cc = 0
DO i = 1, my_nbrs
   k = few_entries_from_this_neighbour
   CALL MPI_IRECV(recv_array(cc+1:cc+k),k, MPI_REAL8, source(i), tag,
MPI_COMM_WORLD, request2(i), ierr)
   DO j = 1, k
       received_data(j)   =    recv_array(cc+j)
   ENDDO
   cc = cc + k
ENDDO

After the above MPI_WAITALL.

Means that,
send_array for all neighbours will have a collected shape:
send_array = [... entries for nbr 1 ..., ... entries for nbr 1 ..., ..., ...
entries for last nbr ...]
And the respective entries will be send to respective neighbours as above.


recv_array for all neighbours will have a collected shape:
recv_array = [... entries from nbr 1 ..., ... entries from nbr 1 ..., ...,
... entries from last nbr ...]
And the entries from the processes will be received at respective
locations/portion in the recv_array.


Is this scheme is quite fine and correct.

I am in search of efficient one.

Request for help.


With best regards,
Amjad Ali.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20091110/adf0f1fc/attachment.html>