[Beowulf] Cheap SDR IB

[Jeffrey B. Layton]

Mark,

Thanks for being the knucklehead that allows me to respond to
you and a bunch of other knuckleheads :)

>> Just in case you've missed the announcements:
>> http://www.clustermonkey.net//content/view/222/1/
>
> I'm always happy about new levels pricing agression, but I'm
> a bit puzzled about for what kind of workloads this will matter.
>
> whenever I ask about IB bandwidth, people always point fingers at 
> weather codes, which apparently are fond of doing the transpose
> in multi-dimension FFT's using all-to-all.  while convenient, this
> seems a bit silly, since transpose is O(N) communications, not O(N^2).
>
> higher bandwidth/node also makes sense if you're configuring fairly
> fat nodes (many cores, probably also lots of ram).  but if you do that,
> you also amortize the networking, so a cheaper IB setup matters less.
>
> perhaps there are some extremely file-IO intensive workloads that can 
> sustain ~1 GB/s, but I'd expect them to require some hefty fileserving 
> hardware, which would also hide the IB cost.
>
> IB for gaming?  I have one ratio: 1e-1/3e-6.  that's human reaction
> time versus IB latency.

Mark - I know these comments are not directed at me per say, but at the
community in general. My response is - test your own applications and
then determine their characteristics and let us all know about what you
have learned. I'm sure everyone is dying to learn.

But, let me also say, that extrapolating characteristics from one code to
other codes that do similar things is dead wrong. It depends on the quality
of the code (as always :)  ).

> also, I think it's a bit disingenous to use 10G Chelsio TOE to compare,
> rather than 10G Myri which is cheaper and faster.  also:
> http://www.chelsio.com/sandia_benchmark_tech.html

Sigh... I've got a couple of emails from people on this. In general the
emails revolve around a single thing "I don't like your numbers!" and
some of them seem to come from vendors. I won't get into full rant mode
on this one (no one would read it anyway), but let me just say, that
the numbers in the table are taken from a table that is about a year old.
I picked just a few numbers that I thought were worthwhile rather
than reproduce the whole table. I'm sorry your favorite interconnect
wasn't in there - I'm sure I'm doing you a great disservice my not
including <insert favorite interconnect here> and the company is mortally
wounded and lief s we know it will collapse and the energy from the
collapse will be great enough to form a black hole centered on the earth
which will cause a chain reaction and the entire universe will be sucked
into it and that's that.

I post what benchmark numbers I have and I always say that the numbers
are probably not comparable between vendors, don't use them to make
a decision, and please test your applications. All of the numbers are at
least a year old, but I haven't had time to look or ask for updated numbers
so I used the best I have clearly saying that the numbers are old.

So, Mark, if you don't like the numbers I invite you, nay I beg you, to
write your own interconnect article with your favorite numbers or
interconnect in there. I'm sure we would all LOVE to hear what you say
(that is said in a serious tone, not condescending like the rest of this
email).

> finally, how the heck do you make Gb as slow as 120 us?

Those numbers are from some old tests. The NICs were pretty
crappy to be honest. But it does represent an upper bound. But
then again, as I said before, please post your own numbers to
either this list (please be sure to include copious notes as I'm sure
some knucklehead will chime in and complain that your numbers
stink and that you should do the test a completely different way.
It happens all of the time). Even better, how about writing something
for ClusterMonkey? The site is for the community and we make
no money what so ever from it. So giving back to the community
is a worthwhile thing (IMHO) when you don't agree with anything.

Enough of my rant - I could go on for days and days about the crap
we see as authors trying to help the community and help people.
It's truly amazing. And ranting is not the point of this list :)
>
> -Mark "not actually anti-IB" Hahn.
How about just "anti-everything" That's seems much more appropriate.

Jeff