Archives

- Beowulf
- Beowulf Announce
- Scyld-users
- Beowulf on Debian

# [Beowulf] bandwidth: who needs it?

Many of your questions may have already been answered in earlier discussions or in the FAQ. The search results page will indicate current discussions as well as past list serves, articles, and papers.

Search

Philippe Blaise philippe.blaise at cea.fr
Fri Oct 22 00:41:58 PDT 2004

```The naive formula

T_size = T_0 + size / max_bandwidth

for
size1/2 = T_0 * max_bandwith
gives

T_size1/2 = 2 * T0

which is a characteristic message length : you reach (more or less) half
the bandwith,
and it takes 2 *  latency seconds to send / recv the message.

For example, with T_0 = 5 usec and max_bandwith = 400 or 800 MB/s you obtain

T_size1/2(5, 400) = 5 * 400 = 2 kB
T_size1/2(5, 800) = 5 * 800 = 4 kB

Phil.

Richard Walsh wrote:

>Greg Lindahl wrote:
>
>
>
>>>do you have applications that are pushing the limits of MPI bandwidth?
>>>for instance, code that actually comes close to using the 8-900 MB/s
>>>that current high-end interconnect provides?
>>>
>>>
>>Bandwidth is important not only for huge messages that hit 900 MB/s,
>>but also for medium sized messages. A naive formula for how long it
>>takes to send a message is:
>>
>>T_size = T_0 + size / max_bandwidth
>>
>>For example, for a 4k message with T_0 = 5 usec and either 400 MB/s or
>>800 MB/s,
>>
>>T_4k_400M = 5 + 4k/400M = 5 + 10 = 15 usec
>>T_4k_800M = 5 + 4k/800M = 5 +  5 = 10 usec
>>
>>A big difference. But you're only getting 266 MB/s and 400 MB/s
>>bandwidth, respectively.
>>
>>Of course performance is usually a bit less than this naive model. But
>>the effect is real, becoming unimportant for packets smaller than ~ 2k
>>in this example. The size at which this effect becomes unimportant
>>depends on T_0 and the bandwidth.
>>
>>
>
>The above also makes a point about a mid-range regime of message sizes
>whose transfer times are affected ~equally by bandwidth and latency
>changes.  Halving the latency in the 4K/800M case above is equivalent
>to doubling the bandwidth for a message of this size:
>
>  T_4k_800M. = 2.5 + 4k/800M  = 2.5 +  5.0 =  7.5 usec
>  T_4k_800M  = 5.0 + 4k/800M  = 5.0 +  5.0 = 10.0 usec
>  T_4k_1600M = 5.0 + 4k/1600M = 5.0 +  2.5 =  7.5 usec
>
>For a given interconnect with a known latency and bandwidth there is
>a "characteristic" message size whose transfer time is equally sensitive
>to perturbations in bandwidth and latency (latency and bandwidth piece
>of the transfer time are equal).  So, for an "Elan-4-like" interconnect
>characteristic message length would be 1.6k:
>
>  T_4k_800M   = 1.0 + 1.6k/800M   = 1.0 + 2.0 = 3.0 usec
>  T_4k_800M   = 2.0 + 1.6k/800M   = 2.0 + 2.0 = 4.0 usec
>  T_4k_1600M  = 2.0 + 1.6k/1600M  = 2.0 + 1.0 = 3.0  usec
>
>Messages sizes in the vicinity of the characteristic length will
>respond approximately equally to improvements in either factor.
>Messages much larger in size will be more sensitive to bandwidth
>improvements in an interconnect upgrade while message sizes much
>smaller will be more sensitive to latency improvements in an upgrade.
>
>One might argue that bandwidth actually matters more because message
>sizes (along with problem sizes) can in theory grow indefinitely (drop
>in some more memory and double you array sizes) while they can be made
>only be so small -- this is a position supported by the rate of storage
>growth, but undermined by slower bandwidth growth and processor count
>increases.
>
>I think I will keep my bandwidth though ... and take any off of the
>hands of those who ... don't need it ... ;-) ...
>
>rbw
>
>#---------------------------------------------------
># Richard Walsh
># Project Manager, Cluster Computing, Computational
>#                  Chemistry and Finance
># netASPx, Inc.
># 1200 Washington Ave. So.
># Minneapolis, MN 55415
># VOX:    612-337-3467
># FAX:    612-337-3400
># EMAIL:  rbw at networkcs.com, richard.walsh at netaspx.com
>#         rbw at ahpcrc.org
>#
>#---------------------------------------------------
># "What you can do, or dream you can, begin it;
>#  Boldness has genius, power, and magic in it."
>#                                  -Goethe
>#---------------------------------------------------
># "Without mystery, there can be no authority."
>#                                  -Charles DeGaulle
>#---------------------------------------------------
># "Why waste time learning when ignornace is
>#  instantaneous?"                 -Thomas Hobbes
>#---------------------------------------------------
>
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
>
>

```