[Beowulf] bandwidth: who needs it?

[Greg Lindahl]

> do you have applications that are pushing the limits of MPI bandwidth?
> for instance, code that actually comes close to using the 8-900 MB/s
> that current high-end interconnect provides?

Bandwidth is important not only for huge messages that hit 900 MB/s,
but also for medium sized messages. A naive formula for how long it
takes to send a message is:

T_size = T_0 + size / max_bandwidth

For example, for a 4k message with T_0 = 5 usec and either 400 MB/s or
800 MB/s,

T_4k_400M = 5 + 4k/400M = 5 + 10 = 15 usec
T_4k_800M = 5 + 4k/800M = 5 +  5 = 10 usec

A big difference. But you're only getting 266 MB/s and 400 MB/s
bandwidth, respectively.

Of course performance is usually a bit less than this naive model. But
the effect is real, becoming unimportant for packets smaller than ~ 2k
in this example. The size at which this effect becomes unimportant
depends on T_0 and the bandwidth.

-- greg