[Beowulf] UPS & power supply instability

[Mark Hahn]

> We have a Liebert 600 Series 500kVA UPS feeding two Liebert PDUs.  The 
> PDUs then have a fanout of whips to the computer racks.

the PDU's are just simple networks, not matching transformers or 
harmonic mitigators?  (for our new ~2k cpus machineroom, the local 
physical plant people required us to put in Liebert harmonic mitigators,
even though we told them all PS's would be PFC.  from HP, if that matters.)

> The UPS voltage in/out and the PDU voltage in is 480V 3ph.  The PDU out 
> is 208V 3ph +neutral, 120V wrt neutral.  The whips are 5-conductor 
> (3ph, ground, neutral), and they feed APC AP7960 switchable rack PDUs.  
> The computers are fed 120V from the AP7960s.

it shouldn't be relevant, but did you choose against 208 to the nodes 
for a reason?  (nearly everything is auto-ranging nowadays, and tends
to run a little more efficiently at 208).

> I've balanced the loads on the three phases about as well as possible.  
> We still have neutral current, about 1/3 to 1/2 the magnitude of any of 
> the per-phase currents.

yow!  isn't that very high?  we had an anti-neutral-current squad on 
campus earlier this year, and they freaked out over our old machineroom
which had neutral that was about 10% of the others...

> The problem is this: We can fire up our cluster to about 40% of maximum 
> load and everything is fine.  But if we go over some threshold right 
> around 40% of max, the output currents from the PDUs go unstable.  It's 

"fire up" means power on at the same time?  what happens if you sneak up
the load (say, one node per minute to be conservative.)?  I'm wondering
whether part of your problem is inrush/spinup load.

> This instability only happens when the UPS is online.  If we put the UPS 
> in bypass, we can go up to around 70% of max load with no instability 
> (all computers on but idling in the OS; we haven't tested all nodes at 
> 100% CPU yet).

ouch!