[Beowulf] How to configure a cluster network

[Patrick Geoffray]

Hi Jan,

Jan Heichler wrote:
> 1) most applications are latency driven - not bandwidth driven. That 
> means that half bisectional bandwidth is not cutting your application 
> performance down to 50%. For most applications the impact should be less 
> than 5% - for some it is really 0%.

If the app is purely latency driven, bandwidth (link or bisection) is 
indeed irrelevant. However, don't underestimate the impact of contention 
on collective communication: once you exceed the internal buffering in 
the crossbars, you will have back-pressure. Typically, each crossbar 
port can buffer in the order of 1-10K these days. So, the larger the 
message size for the collective and the larger the communicator, the 
greater the need for effective bisection. At this scale (ie 50 nodes), I 
agree it's not that important, unless you are bandwidth bounded to begin 
with.

> 2) Static routing in IB networks limits your bandwidth for many of the 
> possible communication patterns anyway. For completely random 
> communication it was like below 50%. So you buy a IB fabric with full 
> bisectional but can't use it anyway - reducing the bisectional bandwidth 
> is not impacting that much anymore (as far as i understood most whitepapers)

With static routing on Fat Tree or Clos and pseudo-random traffic (ie 
real world), you waste ~50% of the bisection you have (actually, the 
more hops the more waste, but it's not linear). So, if you start with 
half the theoretical bisection, your effective bisection will roughly be 
a quarter of that.

Patrick