disadvantages of a linux cluster

[Robert G. Brown]

On Mon, 11 Nov 2002, Guy Coates wrote:

> >Anyone exploring high FLOPS/watt clusters yet?
> 
> We currently have a 768 node cluster RLX cluster in production. We were
> low on power capacity and physical space in our machine room so it made
> sense to pay the premium for the high density. When we did our evaluation
> we ended up with a test metric of BLASTs/Watt/Pound.
> 
> 24 blades (800Mhz PIII, 2x40Gig drive, 3x100Mb ethernet per blade) fit
> into 3U and consume ~840 watts. Compaq/HP, IBM and Fujitsu also offer high
> density/low power blade solutions as well with various GigE/myrinet
> networking options should you require them.
> 
> What solution you end up with depends on how much extra you want to spend
> on hardware in order to save on space and power.

Just for the record, how much did this cluster cost?  Or at least, how
much does a 3U with 24 blades cost?

I'd also be very interested in seeing the actual numbers for the best
power/bogomips densities you could get with Intel and/or AMD.

Offhand it looks like you're getting maybe 2x the Watts/U one can get
with most Intel casings, and maybe 2x the bogomips/MHz as well (e.g.
dual 1.4 GHz PIII's in 1U is 2800 MHz/U or a bit less than half your
6400 MHz/U.  So you don't seem to be getting a lot more MHz/Watt,
although you can certainly pack more MHz and Watts per U (with the
accompanying problem in shedding heat from an even higher power density
than one has in current 2U Intel duals). 

If total power capacity and/or A/C is your true limiting feature (as
opposed to physical space), it isn't clear that you win, especially when
you have to contend with Amdahl's law (across more processors), pay a
lot more money for the hardware up front, have more to fail in a locally
much hotter environment, and more to manage.  Maybe, anyway -- I don't
really know how management of 24 800 MHz blades in 3U compares to
managing the 8 2400 MHz P4's in 8U that would replace them.

This isn't intended to be flames or critical, BTW -- I'm just trying to
play with the numbers without really having the numbers.  Feel free to
correct anything.

  rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu