MPI Send/Recv time is 3 times netpipe time

[Joseph Mack]

I compared the netpipe two way times for packets upto 1Mb with the
time for two way MPI-Send/MPI-Recv of messages of the same size 
(1..10^6 bytes). This was done for 

o a master process and 1-3 slave processes all running on the
same 2-way SMP machine (ie upto 2 processes on each CPU - tcpip
packets all internal to the SMP node).

o a master and 1-3 slaves processes all on different machines 
(1 process on each CPU, ethernet packets over 100Mbps ethernet).

For the MPI-Send/Recv, the master allocated a 1Mbyte array of char which 
was assumed to be abitary garbage in memory. The first n bytes of it
(for n = 1..1Mbyte) were Send/Recv to a slave, the slave would do the 
complementary Recv/Send and the returned message was compared to the 
original (no errors were found).

For more than 1 slave process the time to Send and Recv the messages
at the master was approximately proportional to the number of slaves so
the time for each message send/recv was assumed to be constant.

The times for the packet returns was longer when the packets were
sent between two different nodes compared to between processes running on
one SMP node. However I found that the time for the MPI message send/recv 
was about 3 times that for the same size packet send/recv by the netpipe 
test for the whole range of message sizes (1byte .. 1Mbyte) in both
setups.

Is this ratio of 3::1 about right?

Why is it 3 times longer? It looks like a two more events which takes the same
time as the ethernet transmission are required. Does something happen at both the
send and recv end of the MPI message that takes the same time as the ethernet
transmission?

Thanks Joe
-- 
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center, 
mailto:mack.joseph at epa.gov ph# 919-541-0007, RTP, NC, USA