[Beowulf] Estimating cluster power consumption

[Robert G. Brown]

On Wed, 14 Dec 2005, r reynoldson wrote:

> Hi all. I'm trying to track down some threads from
> this group that date back to the fall of 2003 (Sept -
> Nov). These don't seem to be in the Beowulf mailing
> list archive. In particular, I read in Jeff Layton's
> "Cluster Environments" article
> (http://www.clustermonkey.net//content/view/40/32/)
> that some cluster power considerations were discussed
> in these threads.
>
> Alternatively, if someone here has some insights and
> can spare a few moments, I have a few questions.

Sure.

> First I should say that I just have a little
> experimental cluster (5 nodes including master) right
> now, but have an opportunity to more than double this
> size very cheaply. The compute nodes are a mixture of
> 366 - 450 Mhz boxes, 128 - 256 Mb ram. Currently they
> have hard drives, but I'm seriously considering going
> diskless (for the learning experience, amoung other
> things). Also, the compute nodes are just boxes (no
> monitor, keyboard, or mouse). The master node has an
> 1800 AMD cpu, 512 ram, and 60 Gb hard drive. I'll be
> mainly using the cluster for number crunching
> experiments (primality testing, various integer
> factorisation implemetations, etc). A typical cluster
> run could last anywhere from a day to several weeks
> (possibly months?).

Sounds not unlike my home cluster except for node speed.

> My house, which contains my cluster room, is rather
> old. Looking at my breaker panel, it looks like my
> cluster room shares a 15 amp breaker with an adjoining
> room (sad, I know).
>
> Questions
> 1) About how many of the above mentioned nodes would I
> be able to safely run?

If you buy a device called a kill-a-watt (available from Amazon and a
zillion other places for $30 shipped) you can answer this precisely by
measuring their loaded and unloaded power draw.  Strongly recommended.
Don't build a cluster without it.

The answer will APPROXIMATELY be -- estimate 100 watts/node, so four
nodes = 400 watts.  Estimate an extra 100 watts for your master (total
600 watts).  Add 100 watts just because.  So you're using about half of
the circuit capacity so far.

Now look around at the OTHER loads in the two rooms.  An electric light
is typically 60-100 watts (unless you use compact fluorescent bulbs,
also recommended as they run COOLER and SAVE ENERGY AND THEIR COST IN
MONEY several times over their expected 5+ year lifetime).  An electric
heater is a no-no -- 500+ watts all by itself.  Ditto wall-unit ACs --
you'll have to directly measure draw with your kill-a-watt if you use
one to keep your cluster room cool.  USUALLY a "bedroom" or "living
room" will draw around 200-400 watts at night when 2-3 incandescent
lights are on, the stereo is plugged in and on, the TV is playing a DVD,
your electric massage chair is running.  You can cut this to 70-200
watts by using compact fluorescents.

You're interested in peak draw, since popping a breaker in mid
computation sucks.  So estimate highball -- the probable peak load per
room.  Let's assume that you don't have stereos, TVs, and that you have
three CF lights in each room -- then your load is likely to be no more
than 200 watts/room or 400 more watts.  700 plust 400 = 1100, so you
could almost certainly add four more nodes DEPENDING ON THE ACTUAL DRAW
OF THE NODES WHICH CAN VARY CONSIDERABLY DEPENDING ON WHAT YOU GET with
a margin of roughly 100 watts before blowing a breaker if you should try
to e.g. boot all of the nodes at once with all the lights in both rooms
on and that shiatsu chair running.  This is an estimate -- you might be
able to squeeze in more if your nodes are only drawing 80 watts peak
(they might, they are pretty old/slow and hence relatively cooler);
fewer if your new high-clock nodes draw 150 watts loaded (quite
possible).  Your kill-a-watt and simple addition permit you to answer
the question quite precisely.

Depending on how your house was built, BTW, it may not cost TOO much to
run a dedicated 20 amp circuit to your cluster room.  As in maybe
$100-200.  Way less if you do it yourself, if you CAN do it yourself.
In a garage that contained the breaker box (or into a room right NEXT to
the breaker box where you can easily go through the wall and pull a
piece of wire or from a service that feeds in through a basement with
clearly visible overhead wires goine up through the floor into wall
spaces, on a service with surplus capacity, in possession of a good book
or prior knowledge of the skills required, you could DIY for about $50
-- a piece of wire, a receptacle box and receptacles, a breaker, some
staples.  $70 if you buy a howto book.  Do NOT do this if you aren't
completely comfortable with tools and electricity and if you cannot
religiously follow instructions and understand WHY you are doing what
you do.

> 2) Is there any chance of a fire hazzard if the
> breaker is overloaded or will the breaker just trip?

Always a chance, but if your house is correctly wired with adequate
materials and passed inspection and his correctly breakered, the breaker
will just trip and the probability of fire is very, very low.  If your
house is 100 years old, has a 200 foot long run of two separate 14 gauge
asbestos insulated wires running naked through holes in your very dry
old house's wall studs, where mice have chewed off wood shavings into a
tinder-dry nest that happens to cover a wire and where their activity
has rubbed the wire insulation into dust and thinned the wire itself out
into a "hot spot" of higher resistance than spec (and given the little
rodents mesothelioma in the process, serves them right), well, sure,
there's a hazard there.  Maybe even if the breaker does NOT trip from an
actual overload, as the "overload" that can cause a fire might be lower
than it "should" be.

> 3) How does one estimate how many nodes they can
> safely run?

Add up power draw of everything on the circuit, and compare to 1500
watts is a good first estimate.  You can do better if you learn a bit
about rms vs peak power, power factors of switching power supplies and a
few other things, but they are typically going to be 10-20% effects at
most and I'd recommend maintaining at least a 10% margin under peak
capacity anyway.

> 4) Would running the cluster diskless help much with
> power consumption? What about no CD and floppy drive?

Yes -- by as much as 100 watts or even a bit more over 8-10 nodes.  The
strippeder the better, as everything you put into the system draws at
least some power at least some of the time and it is PEAK draw that
blows the breaker (or starts the fire that will burn those pesky little
rodents -- and your house -- down).

> 5) How much juice should be flowing to the room to run
> a cluster of 16 such nodes?

Depending on what "such nodes" means, somewhere between 1000 and 3000
watts.  A conservative estimate would be somewhere between 10 and 30
amps, if you like.  With electric lights ALSO on the circuit, I'd worry
about fitting them all onto a single 20 amp circuit and would GUESS that
you'll be very squeezed on a 15 amp circuit.  You might get them to run
on a single 20 amp circuit if you didn't also run anything else e.g.
lights or toaster ovens.  I've personally measured node draws from about
70-80 watts (depending on state) to well over 200 watts for pro-grade
dual CPU, feature loaded nodes running an intensive computation.  Noting
that power draw IS a weak function of system computational load (weak in
that there is a base consumption for an idle system that can be 70-90%
of the draw of the fully running system already).

> 6) What's the least expensive way of getting more
> electricity to the room? I can't do it myself (at
> least I think I can't). Which is the more desirable --
> increasing the existing breaker amperage (if possible)
> or getting a second breaker box (or replacing the
> existing box with a bigger and better box)? The house
> has copper wiring in the basement, which may affect
> the cost (I also live in Canada too, eh!).

As I said above, the least expensive is DIY.  I have extensive
experience wiring and teach electrodynamics (and so understand how Mr.
Electricity Works well enough to teach electricians:-).  I can also go
to the web and read about code and so on -- Google is your friend!.  If
I need(ed) to I can(could) buy books at Home Depot that lead me through
adding a simple circuit step by step.  So if my house had copper wiring
in a basement, and my cluster were on the floor right above the basement
(so I could easily drill a hole from the basement up into the space
between walls) and if my breaker box had spare capacity, I would most
definitely do it myself for $50 -- if I were feeling frisky I might even
run in TWO home run 20 amp circuits for good measure, and leave the room
circuit alone to run the lights and so on.  Can't have too much power
capacity, I always say.  Even if I were planning to use a 15 amp breaker
I'd use 12 gauge wire, BTW -- thicker wire is lower resistance and
cooler wire and safer wire.  If 10 gauge weren't expensive, a bloody
pain to bend around and too thick for most receptacles, I'd even use
that (and have, on a few occasions).

However, in this case it ALSO wouldn't be too expensive to have a real
electrician do it -- it shouldn't take them a full hour of time as
described, at maybe $70-100/hour depending on whether or not they have a
helper (helper costs more but SHOULD reduce billed time to half an hour
as the electrician can do the breaker box end while the helper does the
receptacle end).  Might or might not need inspection before energizing,
don't know canada law.  DIY would technically need it also, if so,
although you might -- ahem -- "forget" to have it done at some risk to
your person and your property.

If your cluster room is on the second floor, they (or you) will have to
fishtape the wires up between floors and cut out some drywall to be able
to do the drilling and so on.  If they can go through a closet (often
possible) it might still be cheap enough; otherwise it might start get
up to $300-400 US to put in a couple of circuits (where the second
circuit is basically at cost of the parts, since the same amount of
labor more or less can do two as easily as one).

If you have to upgrade your actual service, add order of $1000 and a lot
of pain, and do NOT try to do this yourself -- it is one thing to work
from a well-installed circuit box with a master breaker to de-energize
the whole thing from the incoming wires down (so you can safely touch
and work on the breaker buses).  It is another to mess with the incoming
supply lines, and besides, that DOES have to be inspected no fooling and
only a fool would try to skip that step for liability and practical
reasons.  Like not dying or watching the entire wall where the breaker
box vaporize in a flash of metal vapor when the box itself is energized,
blowing the breaker on the supply tranformer (if you are lucky), and
getting into a LOT OF TROUBLE if you survive.

> 7) What are some good references to answer such
> questions? I'd even spend money on a book, if it were
> recommended highly enough.

Lots of web references.  My personal favorite is:

   http://www.faqs.org/faqs/electrical-wiring/

(parts one and two) but GIYF.

Also lots of howto books at your favorite hardware chain store.  Many
with nice pictures and entirely adequate instructions for installing a
simple receptacle circuit safely.  None of these will help with cluster
wiring per se, but for a home cluster you just don't need to mess with
the nuances of switching power supplies.  Just install an overcapacity
of at least 20% and do NOT let a commercial electrician install three
different circuits with distinct phases to the space with a shared
neutral line -- make sure that they understand that the application is
computers with a presumed nonlinear load.  Don't worry about
not-understanding what I just said, either -- if you do your own wiring
you would never do this anyway and don't even know (or need to know)
what "phase" is.

   HTH,

     rgb

>
> Thanks in advance
> Rob
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu