[Beowulf] Cheap SDR IB

[Mark Hahn]

>> whenever I ask about IB bandwidth, people always point fingers
>> at weather codes, which apparently are fond of doing the transpose
>> in multi-dimension FFT's using all-to-all.  while convenient, this
>> seems a bit silly, since transpose is O(N) communications, not O(N^2).
>
> Mark,
>
> interconnect does matter.

I did not claim the opposite - I said that for small, cost-sensitive
clusters, it would be unusual to need IB's advantages (high bandwidth
and latency comparable to other non-Gb interconnects.)

in particular, I'm curious about the conventional wisdom about weather codes
and bandwidth.

> Here is a solid benchmark using WRF, 128 cores, 
> Woodcrest 3.00GHz. System spec can be found at 
> http://www.spec.org/mpi2007/results/res2007q4/mpi2007-20071013-00029.html
>
> Scali MPI Connect 5.6.2 using IB (IB as specified in the link above):
> Success 127.wrf2 base mref ratio 21.03, runtime 370.653066

I was curious about this: you only used one DDR port; was that because
of lack of switch ports, or because WRF uses bandwidth <= DDR?

> Scali MPI Connect 5.6.2 using Gbe (Broadcom NetXtreme II BCM5708 1000Base-T 
> (B2), Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.3.29, 9k MTU, 
> Switch? Jeff, can you fill in here, the system should be familiar to you):
> Success 127.wrf2 base mref ratio=4.82, runtime=1618.248048
>
> That's a pretty decent advantage to IB, isn't?

sure, and these are very fat nodes for which a fat interconnect is
appropriate for almost any workload that's not embarassing.  but really
I wasn't suggesting that plain old Gb (bandwidth in particular) was
adequate for all possible clusters.  I was questioning whether IB was 
a panacea for small, cost-sensitive ones...