Fwd: [Beowulf] Re: Why is communication so expensive for very small messages
j.boyle at manchester.ac.uk
Thu Apr 26 03:54:55 PDT 2007
Thanks for the reply.
I tried reducing the number of loops from 5000, I went as low at 7, and still
measured times of 1ms per communication. Below 7 loops I measured a time of
zero, but MPI_Wtime() only has a resolution of 4ms.
I wasn't sure what you meant by handling the return status.
I changed the code to check the returned error value, but find no errors.
And I find putting MPI_Barrier(MPI_COMM_WORLD) after every communication
reduces the time per communication from 1ms to 70microseconds.
---------- Forwarded Message ----------
Subject: [Beowulf] Re: Why is communication so expensive for very small
Date: Tue, 24 Apr 2007 12:30:03 -0700
From: "David Mathog" <mathog at caltech.edu>
To: beowulf at beowulf.org
Jonathan Boyle <j.boyle at manchester.ac.uk> wrote:
> I apologise if this is a naive question, but I'm new to this world of
> I'm using C++/mpi, to get a feel for communication costs I ran tests
> mpptest and my own programs.
> For 2 processor blocking calls, mpptest indicates a latency of about 30
> However when I measure communication times in my own program using a
> start = MPI_Wtime();
> for (unsigned t=1; t<=5000; t++)
> if (my_rank==0)
> MPI_Send(data, size, MPI_INT, 1, tag, MPI_COMM_WORLD);
> MPI_Recv(data, size, MPI_INT, 0, tag, MPI_COMM_WORLD, &status);
> end = MPI_Wtime();
> for size>=4, I get a latency of about 30 microseconds as expected,
> size<4, communication costs increase massively, and latency now
appears to be
Odd. Could be a couple of things going on.
1. By pure bad luck dropping size 4 -> 3 (for instance) may be the
point at which your network saturates. That is, the slightly smaller
packets may be generated slightly faster so that you hit a bottleneck
(max packets per second inbound would be my guess, I've hit that limit
before) and so you see a nonlinear response. You can test this by
dropping the 5000 to 100 or so, so no matter how fast they go,
they don't max anything out.
2. Could be a bug in the mpi implementation you are using.
I've never written anything for MPI in C++, is the return status
being handled somewhere? I don't see it in this code fragment but
C++ is notorious for putting side effects in another section of code,
far far away. If at all possible the status values should be
checked. Sooner or later one of these functions is going to
come back with a bad status and your program should handle that.
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
More information about the Beowulf