[Beowulf] Why is communication so expensive for very small messages?

[Mark Hahn]

> for size>=4, I get a latency of about 30 microseconds as expected, however for
> size<4, communication costs increase massively, and latency now appears to be
> 1ms!

my guess would be some form of interrupt/message coalescing.

> Secondly, can anyone suggest what's going on, or where I can go for more
> information.

what are your nics?  have you looked at driver-level settings (ethtool, I guess)?