[Beowulf] Re: Estimating cluster power consumption (r reynoldson)

Ed Karns edkarns at firewirestuff.com
Thu Dec 15 14:56:06 PST 2005


On Mr. Reynoldson's questions below and Mr. Lux' responses.

It is advised to run four power line branches into your system room  
with quad "four hole" outlets on each branch (total of 16 power holes  
in 8 outlet sets in 4 utility boxes) and with individual circuit  
breakers on each branch ... Two runs on one phase (2 branches) and  
two runs on the other using 12 or 14 gauge wiring. There are several  
peculiarities about running multiple switching power supplies on a  
single 120 / 220 VAC branch (each system has one). It turns out that  
switching power supplies (as found in standard PC cases) can actually  
"sync up" and cause mysterious crashes, the switching power supplies  
placing a 400 Hertz demand wave on top of the 50 / 60 cycle power  
wave. By connecting half of your load to one phase (one of two  
branches) and half to the other phase on separate circuit breakers,  
the 400 Hertz switching demand load is "spread" over the two phases  
and your overall load is "balanced".

" ... What's the least expensive way of getting more electricity to  
the room? I can't do it myself (at least I think I can't). ..."  ...  
Note: Canadian household power may be 220 VAC, split into two phases  
(in some provinces). Check before you buy anything or "plumb up" your  
power lines to the service panel.

Total operating load = 16 nodes X average 200 watts each = 3200+  
watts. (This is a "running load", not a start up load which could  
easily exceed 5000 watts, even without hard drives and monitors.)

Using four wiring branches, figure 1000 watts per branch = #14 AWG  
wiring. I would use 14 gauge "3-wire w/ground Romex" household wire  
(Red, Black, White & ground) making two home runs, two branches in  
each run, back to the main service panel ... connected to four, 15  
Amp circuit breakers (one for each branch, two red wires, two black  
wires). Plug each pair of breakers into opposite phases in the main  
service panel to balancing the load as per above. (This will give you  
a modest expansion allowance up to maybe 20 to 24 nodes.) Note that 3- 
wire w/ground, 14 gauge wiring is commonly available and relatively  
inexpensive = here in California it is less than US$1.00 per foot and  
non-GFI, 15 Amp breakers are less than US$20.00 each.

Ground loops? Connect all white (common wires) and all ground wires  
in the main service panel to the central bus bar and make sure you  
have a very good "local ground" in the main service panel. (Watch  
out! ... ZZZZZZTTT = tag, you've been hit! and are now headed for the  
hospital ... unless you turned off the main breaker before you opened  
the service panel.)


" ... Is there any chance of a fire hazzard if the breaker is  
overloaded or will the breaker just trip? ..." Yes! ... but if you  
have a qualified local electrician examine your work before you crank  
it up, there should be no problem found.

" ... Would running the cluster diskless help much with power  
consumption? What about no CD and floppy drive? ..." ... Yes, but  
only to a limited extent. Trying to micro manage your power usage by  
removing drive mechanisms will be counter productive in the long run.  
Go ahead and leave the hard drives, etc., in the systems. The amount  
of power saved per node is less than 5% ... and being able to boot  
and run each system individually from its own hard drive will save  
you a world of time and possible frustration.

" ... What are some good references to answer such  
questions? ..." ... make good friends with a qualified electrician.

Mr. Lux: " ... so, you can run all 5 boxes on the 15A  
circuit ..." ... Yes, of course. This may however stretch the limits  
of a many breakers. Five nodes X 200 watts per = 1000 watts total,  
minimum ... and assumes that the internal switching power supplies  
will never "sync up" making momentary power demands of 2000 or more  
watts every few cycles. I would not recommend more than four boxes  
per 15 Amp circuit breaker.

Mr. Lux" " ... Figure 200W/node (for lack of a better estimate) =  
3200W.  3200W/110V = 29Amps  29A * 1.2 (design capacity factor) =  
branch circuit capacity of 34 A. ..." ... In Canada with 220 VAC  
power, a can of worms opens up ... dealing strictly with Watts and  
ignoring voltage considerations, you should not try to stress the  
circuit breakers beyond 50 to 60 % of rating = use a 2.00 "design  
capacity factor" and everything will run cool, system nodes, wiring  
and circuit breakers = greatly reduced fire hazards all around =  
greatly improved reliability of the whole system.

Mr. Lux: " ... 6) What's the least expensive way of getting more  
electricity to the room? ... Extension cords from another branch  
circuit <grin>. ..." ... Ah, yes, and a great big grin it is, too ...  
Kids, don't try this at home. Multiple extensions cords are  
absolutely forbidden in every server farm I have ever seen. Imaging  
that you are close to the end of your program run and you invite a  
few friends over to celebrate ... and one of them decides to play a  
little prank and pull an extension cord plug to make room for the wet  
bar ... Ah, yes ... big grins all around. Besided, I was lead to  
believe that fire hazards were of some modest concern =  no  
extensions cords allowed.


Ed Karns
FireWireStuff.com
(A member of IBEW, Journeyman & Commercial Electrical Contractor  
since 1975)



On Dec 15, 2005, at 7:16 AM, beowulf-request at beowulf.org wrote:
>    7. Estimating cluster power consumption (r reynoldson)
>    8. Re: Estimating cluster power consumption (Jim Lux)
>
> Message: 7
> Date: Wed, 14 Dec 2005 17:28:21 -0800 (PST)
> From: r reynoldson <rreynoldson_ng_mail at yahoo.com>
> Subject: [Beowulf] Estimating cluster power consumption
> To: beowulf at beowulf.org
> Message-ID: <20051215012821.15219.qmail at web54108.mail.yahoo.com>
> Content-Type: text/plain; charset=iso-8859-1
>
> ...    First I should say that I just have a little
> experimental cluster (5 nodes including master) right
> now, but have an opportunity to more than double this
> size very cheaply. The compute nodes are a mixture of
> 366 - 450 Mhz boxes, 128 - 256 Mb ram. Currently they
> have hard drives, but I'm seriously considering going
> diskless (for the learning experience, amoung other
> things). Also, the compute nodes are just boxes (no
> monitor, keyboard, or mouse). The master node has an
> 1800 AMD cpu, 512 ram, and 60 Gb hard drive. ... A typical cluster
> run could last anywhere from a day to several weeks
> (possibly months?).
>
> My house, which contains my cluster room, is rather
> old. Looking at my breaker panel, it looks like my
> cluster room shares a 15 amp breaker with an adjoining
> room (sad, I know).
>
> Questions
> 1) About how many of the above mentioned nodes would I
> be able to safely run?
> 2) Is there any chance of a fire hazzard if the
> breaker is overloaded or will the breaker just trip?
> 3) How does one estimate how many nodes they can
> safely run?
> 4) Would running the cluster disk-less help much with
> power consumption? What about no CD and floppy drive?
> 5) How much juice should be flowing to the room to run
> a cluster of 16 such nodes?
> 6) What's the least expensive way of getting more
> electricity to the room? I can't do it myself (at
> least I think I can't). Which is the more desirable --
> increasing the existing breaker amperage (if possible)
> or getting a second breaker box (or replacing the
> existing box with a bigger and better box)? The house
> has copper wiring in the basement, which may affect
> the cost (I also live in Canada too, eh!).
> 7) What are some good references to answer such
> questions? I'd even spend money on a book, if it were
> recommended highly enough.
>
> Thanks in advance
> Rob
>
> ....

> At 05:28 PM 12/14/2005, r reynoldson wrote: ( James Lux replying)
>>
>> Questions
>> 1) About how many of the above mentioned nodes would I
>> be able to safely run?
>
> It's the rare single box that draws more than 200W (unless you've  
> got quad
> xeons or something) without the monitor. You generally don't want  
> to load a
> branch circuit to more than 80% of the rated capacity (0.8*15 =  
> 12A, here)
> 12A is about 1200W (actually more like 1400, but fudging on the low  
> side
> here is a "good thing").. so, you can run all 5 boxes on the 15A
> circuit.  (on a more practical note, I used to do this very thing,  
> run 5
> 386,486, and Pentium 1 boxes on a single branch circuit in my  
> apartmen, it
> was the LaserJet III that caused the problems, especially if I  
> decided to
> vacuum)
>
>> 2) Is there any chance of a fire hazzard if the
>> breaker is overloaded or will the breaker just trip?
>
> The breaker "should" just trip on an overcurrent.  However, if the  
> breaker
> is a gazillion years old it might not.  Also, if the wall  
> receptacle is old
> and decrepit (most particularly, the wires on the back of the  
> receptacle),
> then you could be drawing a normal amount of current (i.e. 10A),  
> and the
> receptacle will still get hot enough to cause problems.  Unusual,  
> but it
> happens.
>
>> 3) How does one estimate how many nodes they can
>> safely run?
>
> Buy yourself a Kill-A-Watt for $30 and actually measure the power  
> consumed.
> (http://www.efi.org/ is one source, there are others)
>
>> 4) Would running the cluster diskless help much with
>> power consumption? What about no CD and floppy drive?
>
> Every little bit helps
>
>> 5) How much juice should be flowing to the room to run
>> a cluster of 16 such nodes?
>
> Figure 200W/node (for lack of a better estimate) = 3200W.  3200W/ 
> 110V =
> 29Amps  29A * 1.2 (design capacity factor) = branch circuit  
> capacity of 34 A.
>
>> 6) What's the least expensive way of getting more
>> electricity to the room?
>
> Extension cords from another branch circuit <grin>.
>
>
>> I can't do it myself (at
>> least I think I can't). Which is the more desirable --
>> increasing the existing breaker amperage (if possible)
>> or getting a second breaker box (or replacing the
>> existing box with a bigger and better box)? The house
>> has copper wiring in the basement, which may affect
>> the cost (I also live in Canada too, eh!).
>
> A lot depends on what else in the house needs to be fixed when they  
> get in
> to make the changes.  Call your local electrician for an estimate,  
> and they
> can tell you what the tradeoffs are.
>
>> 7) What are some good references to answer such
>> questions? I'd even spend money on a book, if it were
>> recommended highly enough.
>
> Read RGBs online beowulf book at the duke brahma site.  He has a whole
> chapter on infrastructure (power, HVAC, etc) issues.
>
>
>> Thanks in advance
>> Rob
>
> James Lux, P.E.
> Spacecraft Radio Frequency Subsystems Group
> Flight Communications Systems Section
> Jet Propulsion Laboratory, Mail Stop 161-213
> 4800 Oak Grove Drive
> Pasadena CA 91109
> tel: (818)354-2075
> fax: (818)393-6875
>
>
>
> ------------------------------
>
> _______________________________________________
> Beowulf mailing list
> Beowulf at beowulf.org
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
> End of Beowulf Digest, Vol 22, Issue 19
> ***************************************

Ed Karns
IndustrialComponent.com
USBStuff / FireWireStuff / WireLessStuf / FiberStuf ... and much more

http://industrialcomponent.com/contact.html



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20051215/8234db95/attachment.html>


More information about the Beowulf mailing list