<br><br><div><span class="gmail_quote">On 12/18/07, <b class="gmail_sendername">Mark Hahn</b> <<a href="mailto:hahn@mcmaster.ca" target="_blank">hahn@mcmaster.ca</a>
> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> The machines are running the 2.6 kernel and I have confirmed that the max<br>> TCP send/recv buffer sizes are 4MB (more than enough to store the full<br>> 512x512 image).<br><br>the bandwidth-delay product in a lan is low enough to not need
<br>this kind of tuning.</blockquote><div><br>I didn't actually do any tuning, I just checked the max buffer size that the linux auto-tuning can use is sufficient. </div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> I loop with the client side program sending a single integer to rank 0, then<br>> rank 0 broadcasts this integer to the other nodes, and then all nodes send<br>> back 1MB / N of data.<br><br>hmm, that's a bit harsh, don't you think? why not have the rank0/master
<br>as each slave for its contribution sequentially? sure, it introduces a bit<br>of "dead air", but it's not as if two slaves can stream to a single master<br>at once anyway (each can saturate its link, therefore the master's link is
<br>N-times overcommitted.)</blockquote><div><br>I guess I figured that the data is relatively small compared to the bandwidth, whereas the latency for ethernet is relatively high. I also thought the switch would be able to efficiently buffer and forward the data. I am not much of a networking guy (more a graphics guy) so I realize I could be way off base here.
</div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> To make sure there was not an issue with the MPI broadcast, I did one test<br>> run with 5 nodes only sending back 4 bytes of data each. The result was a<br>> RTT of less than 0.3 ms.<br><br>isn't that kind of high? a single ping-pong latency should be ~50 us -
<br>maybe I'm underestimating the latency of the broadcast itself.</blockquote><div><br>This is quite a bit more than a single ping-pong. The viewer sends to the master node (rank 0), and then the master node broadcasts to all other nodes, and then all nodes send back to the viewer node. I don't know if this is still seems high?
<br> </div><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> One interesting pattern I noticed is that the hiccup frame RTTs, almost<br>> without exception, fall into one of three ranges (approximately 50-60,<br>> 200-210, and 250-260). Could this be related to exponential back-off?
<br><br>perhaps introduced by the switch, or perhaps by the fact that the bcast<br>isn't implemented as an atomic (eth-level) broadcast.<br></blockquote><div><br>But the bcast is always just sending 4 bytes (a single integer), and as mentioned above no hiccups occur until the size of the final gather packets (from all nodes to the viewer) is increased.
<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>> Tommorow I will experiment with jumbo frames and flow control settings (both
<br>> of which the HP Procurve claims to support). If these do not solve the<br>> problems I will start sifting through tcpdump.<br><br>I would simply serialize the slaves' responses first. the current design
<br>
tries to trigger all the slaves to send results at once, which is simply<br>not logical if you think about it, since any one slave can saturate<br>the master's link.<br></blockquote><div><br>I still have the feeling that the switch should be able to handle this more efficiently, but since your idea is relatively simple to implement I will give it a try and see what the performance is like.
<br><br>Thanks for your input.<br><br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>regards, mark hahn.<br></blockquote></div><br>