[Beowulf] Cheap SDR IB

[Mark Hahn]

> Just in case you've missed the announcements:
> http://www.clustermonkey.net//content/view/222/1/

I'm always happy about new levels pricing agression, but I'm
a bit puzzled about for what kind of workloads this will matter.

whenever I ask about IB bandwidth, people always point fingers 
at weather codes, which apparently are fond of doing the transpose
in multi-dimension FFT's using all-to-all.  while convenient, this
seems a bit silly, since transpose is O(N) communications, not O(N^2).

higher bandwidth/node also makes sense if you're configuring fairly
fat nodes (many cores, probably also lots of ram).  but if you do that,
you also amortize the networking, so a cheaper IB setup matters less.

perhaps there are some extremely file-IO intensive workloads that 
can sustain ~1 GB/s, but I'd expect them to require some hefty 
fileserving hardware, which would also hide the IB cost.

IB for gaming?  I have one ratio: 1e-1/3e-6.  that's human reaction
time versus IB latency.

also, I think it's a bit disingenous to use 10G Chelsio TOE to compare,
rather than 10G Myri which is cheaper and faster.  also:
http://www.chelsio.com/sandia_benchmark_tech.html

finally, how the heck do you make Gb as slow as 120 us?

-Mark "not actually anti-IB" Hahn.

PS: does anyone have first-hand experience with ConnectX performance?