[Beowulf] MIPS/Watt data
James.P.Lux at jpl.nasa.gov
Thu Jun 3 10:06:31 PDT 2004
At 11:24 AM 6/3/2004 -0500, Brian D. Ropers-Huilman wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>IBM's BlueGene systems are based on similar architectures, 500MHz PowerPCs,
>full system on a chip design.
>On 2004-06-02 09:57 (-0500), Bari Ari <bari at onelabs.com> wrote:
>] Jim Lux wrote:
>] > If we compare Dhry mips to Bogomips (who's to say if we're not within an
>] > order of magnitude).. A 3.4 GHz P4 turned in 6700 Bogomips. That was
>] > probably around 100W (total guess), for 67 Bogomips/watt.
>] ARM, Mips and SH all beat the pants off x86 in the MIPS/watt and also in
>] the FLOPS/watt department.
>] Nobody seems to be interested in clusters built with these
>Brian D. Ropers-Huilman :: Manager :: High Performance Computing
Not exactly nobody: I'm VERY interested in minimizing joules per
computation, because every joule and watt is precious in space....
Lest you think that my application is peculiar, it really isn't, it's just
an extreme case of the increasing emphasis on total cost of
ownership. Power consumption is going to be more and more important,
especially for clusters with more than 4-5 CPUs: Ballpark it.. If your node
costs $1000/CPU and it burns 100W/CPU. You'll actually need more like
150W/CPU (by the time you pump that heat to the big outdoors). Run that
CPU 24/7 for a year, and you've burned 1.3 MWh, about $250 dollars worth, a
pretty big fraction of the cost of the node (a REAL big fraction if you
amortize over 3 years). Cut the power in half to do your computation, and
you've just effectively bought yourself a bunch more nodes.
There are other capital cost aspects to moving the heat around. Taking a
quick gander at the Grainger catalog... A ductless split system with 23000
BTU/hr capacity runs about $2300 and consumes about 2200W (SEER very close
to 10). 23kBTU/hr is 6.8 kW. Figuring a 50% overall system efficiency, we
get pretty close to 3 Watts/dollar. (We'll also assume that includes
installation costs, etc.) So, to move that 150W out of the room costs
about $50 in capital costs, as well. Could be twice or three times that, or
Before people start beating me up about my pessimistic estimates for power
and HVAC costs, the real point is that reducing the energy consumed to do a
unit of computation is a "good thing". There is a distinctly non-zero cost
to moving the heat away from the CPU to the great outdoors that pervades
every part of a cluster. You need bigger heat sinks or heat pipes and
bigger fans, which makes the chassis bigger and heavier, which means that
the rack has to be stronger, which means that the floor has to be stronger,
and the room probably has to be bigger.
For trivial sized clusters, where you can "fit everything under a desk",
this is less of an issue, but you start to get any size at all, the
infrastructure costs start to come out of the noise level.
I think that in the cluster market, this is where vendors are going to have
to start competing. Clearly, they can't try to make deals on the cluster
software, because (as evidenced by all the discussion on the list of late)
the software is basically low cost. A vendor can sell on "quality of
hardware", but underneath it all, there's only a few mobo manufacturers, so
it comes down to who gets the best quantity discounts, and who has the best
sheet metal fabricators, which is a pretty competitive business. The basic
cost to design, market, and fabricate nodes and racks for a cluster is
pretty much the same for all vendors.
So, how, as a vendor, do you differentiate yourself from all the rest?
(especially if someone wants to make a decision based on quantitative metrics.)
- service: repair frequency and speed, documented low MTBF and MTTR,
etc. - these are tough to quantify in any meaningful way, because clusters
are still essentially "one-off" unique items, so statistics are not
particularly meaningful. This comes down to the fuzzy things like "vendor
- unique value added: Training, preconfigured systems, etc. - again, very
difficult to quantify, because some bright soul at your prospective
customer will come up with the "why should we pay X all that money, when I
can download the software, burn the CDS and get it up and running in my
spare time".. Granted it's bogus, but it happens. A vendor can't prove
that the "do it yourself" approach is actually more expensive, and unless
the customer has built some clusters themselves, they won't know any better.
- lower cost of ownership - something that is readily quantifiable! I know
what electricity costs, what floor space costs, what sysadmin time costs, etc.
James Lux, P.E.
Spacecraft Telecommunications Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
More information about the Beowulf