[Beowulf] MIPS/Watt data

[Bari Ari]

We're looking at building some large clusters now using single and dual 
core 64bit MIPS SOC's:

http://www.pmc-sierra.com/products/details/rm9150/

http://www.pmc-sierra.com/products/details/rm9000x2GL/

Another way to go currently is to use the IBM Blue Logic® SOC:

http://www-306.ibm.com/chips/products/asics/products/soc.html

These two solutions would be much lower Watts and $$/FLOP than x86.

The Via C3 parts are dropping in price but they don't offer much in 
cache sizes and I/O throughput. If your code fits into their cache and 
doesn't need to be passed around with other nodes much they aren't a bad 
option.

-Bari

Jim Lux wrote:

> At 11:24 AM 6/3/2004 -0500, Brian D. Ropers-Huilman wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> IBM's BlueGene systems are based on similar architectures, 500MHz 
>> PowerPCs,
>> full system on a chip design.
>>
>> On 2004-06-02 09:57 (-0500), Bari Ari <bari at onelabs.com> wrote:
>> ] Jim Lux wrote:
>> ] > If we compare Dhry mips to Bogomips (who's to say if we're not 
>> within an
>> ] > order of magnitude).. A 3.4 GHz P4 turned in 6700 Bogomips.  That was
>> ] > probably around 100W (total guess), for 67 Bogomips/watt.
>> ]
>> ] ARM, Mips and SH all beat the pants off x86 in the MIPS/watt and 
>> also in
>> ] the FLOPS/watt department.
>> ]
>> ] Nobody seems to be interested in clusters built with these
>> ] architectures.
>>
>> - --
>> Brian D. Ropers-Huilman  ::  Manager  ::  High Performance Computing
> 
> 
> Not exactly nobody:  I'm VERY interested in minimizing joules per 
> computation, because every joule and watt is precious in space....
> 
> Lest you think that my application is peculiar, it really isn't, it's 
> just an extreme case of the increasing emphasis on total cost of 
> ownership.  Power consumption is going to be more and more important, 
> especially for clusters with more than 4-5 CPUs: Ballpark it.. If your 
> node costs $1000/CPU and it burns 100W/CPU.  You'll actually need more 
> like 150W/CPU (by the time you pump that heat to the big outdoors).  Run 
> that CPU 24/7 for a year, and you've burned 1.3 MWh, about $250 dollars 
> worth, a pretty big fraction of the cost of the node (a REAL big 
> fraction if you amortize over 3 years). Cut the power in half to do your 
> computation, and you've just effectively bought yourself a bunch more 
> nodes.
> 
> There are other capital cost aspects to moving the heat around. Taking a 
> quick gander at the Grainger catalog... A ductless split system with 
> 23000 BTU/hr capacity runs about $2300 and consumes about 2200W (SEER 
> very close to 10). 23kBTU/hr is 6.8 kW.  Figuring a 50% overall system 
> efficiency, we get pretty close to 3 Watts/dollar. (We'll also assume 
> that includes installation costs, etc.)  So, to move that 150W out of 
> the room costs about $50 in capital costs, as well. Could be twice or 
> three times that, or half.
> 
> Before people start beating me up about my pessimistic estimates for 
> power and HVAC costs, the real point is that reducing the energy 
> consumed to do a unit of computation is a "good thing".  There is a 
> distinctly non-zero cost to moving the heat away from the CPU to the 
> great outdoors that pervades every part of a cluster.  You need bigger 
> heat sinks or heat pipes and bigger fans, which makes the chassis bigger 
> and heavier, which means that the rack has to be stronger, which means 
> that the floor has to be stronger, and the room probably has to be bigger.
> 
> For trivial sized clusters, where you can "fit everything under a desk", 
> this is less of an issue, but you start to get any size at all, the 
> infrastructure costs start to come out of the noise level.
> 
> I think that in the cluster market, this is where vendors are going to 
> have to start competing.  Clearly, they can't try to make deals on the 
> cluster software, because (as evidenced by all the discussion on the 
> list of late) the software is basically low cost.  A vendor can sell on 
> "quality of hardware", but underneath it all, there's only a few mobo 
> manufacturers, so it comes down to who gets the best quantity discounts, 
> and who has the best sheet metal fabricators, which is a pretty 
> competitive business. The basic cost to design, market, and fabricate 
> nodes and racks for a cluster is pretty much the same for all vendors.
> 
> So, how, as a vendor, do you differentiate yourself from all the rest? 
> (especially if someone wants to make a decision based on quantitative 
> metrics.)
> 
> - service: repair frequency and speed, documented low MTBF and MTTR, 
> etc.  - these are tough to quantify in any meaningful way, because 
> clusters are still essentially "one-off" unique items, so statistics are 
> not particularly meaningful.  This comes down to the fuzzy things like 
> "vendor reputation".
> 
> - unique value added: Training, preconfigured systems, etc.  - again, 
> very difficult to quantify, because some bright soul at your prospective 
> customer will come up with the "why should we pay X all that money, when 
> I can download the software, burn the CDS and get it up and running in 
> my spare time".. Granted it's bogus, but it happens.  A vendor can't 
> prove that the "do it yourself" approach is actually more expensive, and 
> unless the customer has built some clusters themselves, they won't know 
> any better.
> 
> - lower cost of ownership - something that is readily quantifiable!  I 
> know what electricity costs, what floor space costs, what sysadmin time 
> costs, etc.
> 
> 
> James Lux, P.E.
> Spacecraft Telecommunications Section
> Jet Propulsion Laboratory, Mail Stop 161-213
> 4800 Oak Grove Drive
> Pasadena CA 91109
> tel: (818)354-2075
> fax: (818)393-6875
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf
> 
>