[Beowulf] Re: Why is communication so expensive for very small messages

[David Mathog]

Jonathan Boyle <j.boyle at manchester.ac.uk> wrote:

> 
> I apologise if this is a naive question, but I'm new to this world of 
> beowulfs.
> 
> I'm using C++/mpi, to get a feel for communication costs I ran tests
using 
> mpptest and my own programs.
> 
> For 2 processor blocking calls, mpptest indicates a latency of about 30 
> microseconds.
> 
> However when I measure communication times in my own program using a
loop as 
> follows....
>    
> MPI_Barrier(MPI_COMM_WORLD);
> start = MPI_Wtime();
> for (unsigned t=1; t<=5000; t++)
> {
>  if (my_rank==0)
>  {
>   MPI_Send(data, size, MPI_INT, 1, tag, MPI_COMM_WORLD);
>  }
>  else
>  {
>   MPI_Recv(data, size, MPI_INT, 0, tag, MPI_COMM_WORLD, &status);
>  }
> }
> end = MPI_Wtime();
> 
> for size>=4, I get a latency of about 30 microseconds as expected,
however for 
> size<4, communication costs increase massively, and latency now
appears to be 
> 1ms!

Odd.  Could be a couple of things going on.

1.  By pure bad luck dropping size 4 -> 3 (for instance) may be the
point at which your network saturates.  That is, the slightly smaller
packets may be generated slightly faster so that you hit a bottleneck
(max packets per second inbound would be my guess, I've hit that limit
before) and so you see a nonlinear response.  You can test this by
dropping the 5000 to 100 or so, so no matter how fast they go,
they don't max anything out.

2.  Could be a bug in the mpi implementation you are using.

I've never written anything for MPI in C++, is the return status
being handled somewhere?  I don't see it in this code fragment but
C++ is notorious for putting side effects in another section of code,
far far away. If at all possible the status values should be
checked. Sooner or later one of these functions is going to
come back with a bad status and your program should handle that.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech