[Beowulf] interconnect and compiler ?

[Greg Lindahl]

On Fri, Jan 30, 2009 at 07:07:34AM -0700, Michael H. Frese wrote:

> Johnn Adams said "Facts are stubborn things," and there just aren't  
> enough of them in your example to determine whether bandwidth or latency 
> dominates communication time.

Mark asked for an example, not a research paper. And we were
discussing something other than latency and bandwidth, because these 2
parameters aren't the only fundamental ones for a communications
network.  In the Berkeley "logp" model, for example, processor
overhead and the "gap" betweeen messsages are fundamental parameters.
The InfiniPath chip has a tiny "o" and a negative "g".  As a result,
it can send a lot more small messages than other interconnects, and
this number rises as you add more cores to a system (!). Mark was
wondering when that was important.

Even logp doesn't describe an interconnect that well. It matters how
efficient your interconnect is at dealing with multiple cores, and the
number of nodes. As an example of that, MPI implementations for
InfiniBand generally switch over to higher latency/higher overhead
mechanisms as the number of nodes in a cluster rises, because the
lowest latency mechanism at 2 nodes doesn't scale well.

-- greg