[Beowulf] powering up 18 motherboards

[Robert G. Brown]

On Fri, 18 Feb 2005, David Mathog wrote:

> >I was really hoping a decent $200-300 UPS would come to
> > the rescue here. Oh well.
> 
> APC makes Power Distribution Units that can be set to 
> start loads at fixed intervals.  If you had access to a
> 208V/3 phase line the AP7990 costs about $650, uses that
> as input, and outputs 24 120V sockets.  They make quite
> a few of these PDUs in different configurations
> so maybe you can find one that does what you want?

Which brings to mind a safety as well as a practical point -- if you
indeed DO have three phases of power on the three separate circuits of
the room (not as unlikely as it sounds as in that case they may well
share a single common ground wire) then your power options are a little
different, because then you de facto run at a higher voltage and lower
line current, and can in principle get a small boost in what the lines
can safely tolerate by way of delivered power vs power dissipated in the
supply lines as heat if you use such a unit.

However, this then brings up a nasty, thorny issue concerning the power
factor and current draw pattern of "most" (cheap) PC power supplies.
Switching power supplies that are not power factor corrected tend to
draw most of their current only in the middle third of each voltage
half-sinusoid wave (true fact -- read about it on How Things Work
website or any of several websites that discuss computer room wiring,
some of which -- mirus international? -- are linked to some of my
beowulf pages on brahma).

This means several things:

  a) the peak current draw (for a given average power consumed) is much
higher than you'd expect based on simple RMS considerations -- the power
factor of the load is less than 1, if you know what that means.

  b) this causes higher order harmonics to appear in the voltage/current
curves -- in particular 3*60 Hz or 180 Hz (the "edge" frequency of where
the power draws switch on and off).  These voltage/current ripples may
make it past 60 Hz filters to reach internal components.

  c) three phases can share a neutral because three equal loads with
unit power factor (resistive loads like a light bulb) cause the neutral
current to >>cancel perfectly<<, and it can be shown (consider sin(wt) +
sin(wt + 2\pi/3)) that the neutral current has a strict upper bound in
this case that is within the safe limits of the neutral wire if the
loads on each delivery line are themselves safe.  This is NOT TRUE for
switched power supplies sharing a neutral.  In that case the currents
delivered to the neutral wire by each phase >>add<< instead of
cancelling, in three separate chunks per half cycle.  The neutral
current can actually approach 3I where I is the (average) current being
drawn by any single line (already high relative to RMS expectations
based on an assumption of unit power factor), see above).

This final point is both dangerous and annoying.  It is dangerous
because the neutral line can be carrying enough current to make it much
hotter than permitted by spec assumptions, and if the wiring job is in
any other way marginal, the margins can add and produce a fire (perhaps
during one of those current-draw spikes).  It is annoying because your
CB's will tend to overheat and pop prematurely, your PS's will tend to
overheat and break, your computers will tend to brown out and fail in
mid run due to a voltage ground loop (high backvoltage on your neutral
line relative to prevailing/local/plumbing ground), a condition that is
also actively dangerous on non-ground-fault-protected circuits.  If (as
one might reasonably expect) they are underfusing your line because they
have exceeded the spec length for the gauge of wire that they are using,
then the current carrying neutral ALREADY has a higher resistance than
is technically safe, and all of these conditions are likely to be
exacerbated, possibly into the Danger Zone especially the ground loop
thing.  Shorting that neutral to local building ground might be very
dangerous indeed.

Note that this state COULD ALREADY EXIST with your wiring, and that the
only way to test it is to measure the hot-to-hot voltage between
circuits, that is:

   v_a - v_b = (should be zero, could be 240 or 209 VAC)
   v_a - v_c = ( ditto )
   v_b - v_c = ( ditto )

where v_a is multimeter probe in VAC mode inserted into hot slot of
circuit a, v_b is other probe inserted into hot slot of circuit b.  If
these measurements are all zero, all three of your circuits have the
same phase (and actually could be combined "safely" into a 30 amp
circuit although you should NEVER DO THIS as there is nothing to prevent
somebody at the distribution panel from moving one of the lines onto a
different phase while rearranging things for some other reason so it
isn't at all safe, actually).

If they are all 209, you have three phase (wye) power and the power is
likely being delivered from the distribution panel as a single cable
with five internal wires, three of them insulated carrying one phase
each, a shared insulated neutral, and a bare ground.  If one pair is
zero and the other two are 240, you have two phase power at the
distribution box, and they are either running three separate lines
(likely) or (possibly) lines with four wires, two "hot" and carrying the
opposed phases, one neutral, and one ground and some other circuit in
your apartment is the partner of the odd line out.

They could be running a lower current limit on the lines because they
exceeded the run length for the gauge of wire they used in these cables.
It is certainly cheaper to re-fuse than it is to put additional primary
or secondary panels with thicker wire and/or additional transformers in
locations from which standard wiring can reach and be within code.  A
lot of state codes prohibit this sort of thing and require a building's
wiring to be brought up to code any time anything is renovated, but it
wouldn't surprise me to learn that NYC either makes a general exception
or that individual landlords grease their way to a local exception.

It is perhaps worth your while to figure this out -- I'd certainly want
to know if it were my cluster.  I'd also be CERTAIN to test the phase
per circuit (is the hot wire really hot and not the wire that is
SUPPOSED to be the neutral wire?).  There are some really "interesting"
things that can happen if you cross connect devices in certain ways
between two miswired circuits, where running on each circuit alone is
safe enough in the sense that nothing breaks.  Interesting like
spattering partially vaporized liquid metal is interesting.  You also
very definitely want to learn about the shared neutral thing, if you're
using stock power supplies.  If it is three phase wye, then you could
think about the APC option above and other thing.

Cluster room wiring is nontrivial (even when the "room" is in your
house), and where you are trying to push it to a limit, you're likely
going to have to educate yourself about it.

I think that I put some of this in my online book, in case this is
confusing.  There are also really good discussions of it in the list
archives (where Jim Lux and others made some great contributions).

   rgb

> 
> Another, much cheaper option, would be to set the slave
> node BIOS to use "Wake on LAN" (if it works on your systems)
> and NOT to start on power up.  Then when power came up the
> headnode would boot and the others would just warm up
> enough to listen to their ethernet cards.  I don't
> expect that the start up current going to that mostly off 
> state would be very high, even for 18 computers, since neither
> the disks nor fans start spinning. Once the head node
> comes up you can boot the slaves using etherwake.
> 
> Regards,
> 
> David Mathog
> mathog at caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu