[Beowulf] Why is communication so expensive for very small messages?

[Patrick Geoffray]

Jonathan Boyle wrote:
> For 2 processor blocking calls, mpptest indicates a latency of about 30 
> microseconds.
> 
> However when I measure communication times in my own program using a loop as 
> follows....

If you don't want flow control problems, you need to do a pingpong (each 
node send and recv alternatively) or do explicit flow control using a 
window nad ack message. MPI_Send() can return as soon as the data is 
buffered (ie copied somewhere on the send side), and that is much faster 
than the network itself, so the somewhere will fill up eventually.

Patrick