Need ball park power and cooling requirements

[Robert G. Brown]

On 1 Oct 2002, Jim Meyer wrote:

> On Mon, 2002-09-30 at 14:17, Timothy H. Keitt wrote:
> > I'm setting up lab space and plan to build a small (8-16 node) cluster
> > of dual Xeon nodes (Dell 2650 or equivalent). I need to give the
> > engineers the power and cooling needed in the 6'x8' server room that
> > will house the cluster. Any rules-of-thumb for what's required? Thanks.
> 
> A good friend of mine gave me three magic formulas when I was asked for
> the same information for a new data center to hold ~1600 CPUs worth of a
> render farm. That good friend is now a VERY good friend as it enabled us
> to immediately spot that the contractors proposals were underpowered and
> undercooled. =]
> 
> I must confess that I've used these without fully understanding what
> they represent. I'll add my deductions beneath each formula and invite
> some of the (many) people here who do fully understand them to comment.
> 
> Rough KVA at 3 phase = ((Amps * Volts) * 1.73) / 1000
>   The 1.73 is an approximation of the square root of 3. I don't
>   know why it's there, though, other than a blind guess that
>   it's related to 3-phase power.

On a per-circuit basis (without worrying about the phases) I would
suggest instead:

  a) Determine average power consumed by your systems, per node.  To be
conservative, make a highball estimate.  Let's call this number W (in
Watts = V*A).
  b) W = (1/2) V_max*I_max * PF

where V_max is the peak AC voltage (generally 120V), I_max is the peak
current drawn, and PF is the power factor of the load, which has to do
with whether the voltage and current are in phase.  The factor of (1/2)
comes from the fact that (1/T)\int_0^T sin(\omega t)*sin(\omega t +
\delta) dt is 1/2 when the phase difference between current and voltage
is zero (the average of sin^(\theta) is 1/2).  The power factor comes
from the value of the integral when \delta is NOT zero (voltage and
current are not in phase) and PF <= 1.0.

In the particular case of switching power supplies, the power factor is
< 1.0 not because of a phase difference per se but because of harmonics
at different frequencies in the current, but 0.8 is a not unreasonable
value.  From a) and b) one can compute the maximum CURRENT I_max
drawn per node:

  c) I_max = (2W)/(V_max * PF).  So for a 120W single CPU node, I_max =
240/(100) = 2.4A.  On a 20A circuit (rated in terms of peak current, not
rms current), you could therefore fit about 8 nodes, although 6-7 would
be safer as sometimes (when starting up, when accessing a disk) systems
draw momentary currents in excess of these "steady state" estimates.

Note that if one used power factor correcting power supplies (with PF
\approx 1) you could get as many as ten nodes on the same circuit.

Note also that the >>peak<< current is considerably higher than the rms
>>average<< current because of the power factor and the factor of 1/2 in
the sin^2 average.  The power company has to be prepared to deliver that
peak current, and generally charge you at least partially on the basis
of peak current delivery.

Note that switching power supplies draw their current mostly in the
middle third of each half-cycle in voltage.  This current is delivered
to the neutral wire. Your neutral lines should therefore NOT be shared
between multiple phases (a common practice in wiring where the PF is
expected to be one with no harmonic distortion).

Finally, note that if the actual wires delivering your power and the
return neutrals are long (and have a significant resistance) two things
can occur.  One is that the sharp peak in current draw creates a
significant voltage drop across the supply and neutral lines, which in
turn reduces the line voltage during the middle part of the cycle.  If
you like, the load >>distorts<< the voltage sinusoid by making it more
like a square wave, which can cause odd things to happen to your
equipment if the distortion gets large enough.  Second is that the
voltage drop across the neutral means that the neutral wire is NOT at
zero volts (relative to local ground) at the point where the system is
plugged in.  The chassis of your computer is typically at ground, and
the neutral wire running into your power supply can float a half-volt or
so above ground.  This both produces further oddities in system behavior
and is in fact quite dangerous, as the power line can deliver a LOT of
current across this voltage difference if it should ever be shorted out.
In addition to not being shared, your power lines should therefore be
thick and the neutrals grounded to building steel or other high capacity
ground close to the load.

Do not assume that your contractors know all of these facts.  Ours did
not, and we're STILL trying to get our server room power lines fixed.

> 
> Rough BTU/hr cooling = (((Amps * Volts) * 0.8) / 1000) * 3413
>   The 0.8 is something called "power factor" I think, and largely
>   relates to how efficiently the power is converted and used. I
>   believe the 3413 is just a conversion factor; it takes 3413 BTUs/hr
>   to cool a constant load of 1250VA at 0.8 power factor. I think. =]
> 
> Rough AC Tonnage     = BTUs per hour/12000

It is a lot easier to combine these two computations, especially when
the power factor is irrelevant to the cooling requirements (it increases
the peak current required to produce a given AVERAGE power consumption,
but the AC has to remove the AVERAGE power being consumed, not the peak
VA).  If you do a long and tedious computation that converts a "ton" of
AC (air conditioning that can remove enough heat to melt a ton if ice at
32F in 24 hours, so help me -- God help the evil fiends who came up with
this non-SI-unit abomination:-) into something approximating sane units
(SI) one concludes that a one ton air conditioner removes a teen bit
more than 3500 Joules of energy per second (3500 watts) continuously.

HOWEVER, when computing the wattage that needs to be removed, one has to
remember that heat bleeds into any cool space from any contiguous hotter
place, AND that you need to room to be considerably cooler than ambient
air in most situations.  Building ambient might be (e.g.) 72F but you'd
like your server room to be (e.g.) 60F.  You'd need some AC capacity to
maintain this temperature differential even for an empty room with no
nodes at all.

The >>baseline<< estimate for cooling capacity is therefore based on the
number of nodes.

  d) T = N*W/3500

where T is the number of tons of AC, N is the number of nodes to be run,
W is their average power, and 3500 converts Watts in to Tons.  You can
then apply a fudge factor to come up with your actual requirement:

  d') T' = T*1.2 or T*1.5 or whatever.

The correction should be sufficient to deal with human bodies in the
space (100W each), electric lights in the space (add their wattage),
heat coming in through walls, windows(! -- a window can add LOTS of
wattage in the form of sunlight and conduction), and any other non-node
electrical equipment you might have handy -- monitors, switches, video
games to play while waiting for jobs to run.

Warning/disclaimer:  Although I've been gradually building up the
understanding explicitly represented above for some years now, it could
still have some errors in it (although it almost certainly is, if
anything, too conservative and not too liberal).  I invite comment and
correction from those on the list that are wiser than I.  I'm very
shortly going to go ahead and add the above to my online book so that
this FAQ can be referred to the book in the future.

   rgb

> 
> Hope that helps!
> 
> --j
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu