[Beowulf] Academic sites: who pays for the electricity?

[Jim Lux]

At 04:19 PM 2/17/2005, Jim Lux wrote:
>At 02:54 PM 2/17/2005, David Mathog wrote:
>>At Wed, 16 Feb 2005 19:08:05 +0100 Vincent Diepeveen wrote:
>>
>> > Date: Wed, 16 Feb 2005 19:08:05 +0100
>> > From: Vincent Diepeveen <diep at xs4all.nl>
>> > Subject: Re: [Beowulf] Academic sites: who pays for the electricity?
>> > To: "David Mathog" <mathog at mendel.bio.caltech.edu>,
>> >       beowulf at beowulf.org
>> > Message-ID: <3.0.32.20050216190804.0106fcc0 at pop.xs4all.nl>
>> > Content-Type: text/plain; charset="us-ascii"
>> >
>> > At 08:16 16-2-2005 -0800, David Mathog wrote:
>> > >In most universities services like electricity, water, and
>> > >A/C are paid for by the school.  To do so they take "overhead"
>> > >out of every grant.  Partially as a consequence of this they
>> > >typically have a very poor ability to meter usage on a room
>> > >by room basis.
>> > >
>> > >Now somewhere between the 10 node Pentium II beowulf sitting on
>> > >a lab bench and the 1000 node dual P4 Xeon beowulf in a machine
>> > >room that takes up half the basement the cost of the electricity
>> > >(both for power and A/C) goes from  a minor expense to a major
>> > >one.  Really major. For instance, in that hypothetical large machine,
>> > >at 10 cents per kilowatt hour (a round number), assuming 100 watts
>> > >per CPU (another round number) that's:
>> > >
>> > >  1000  (nodes) *
>> > >     2  (cpus/node) *
>> > >     .1 (kilowatts/cpu) *
>> > >     .1 (dollars/kilowatt-hour) *
>> > >  365   (days /year) *
>> > >   24   (hours/day) =
>> > >-----------------------
>> > >  175200 dollars/year
>> >
>> > Complete academic nonsense calculation. If you use quite some electricity
>> > the electricity gets up to factor 20-40 cheaper. Getting a factor 10
>> > reduction in usage bill is pretty easy if you negotiate properly.

Just where do you live that such negotiations are possible. Here's some 
real numbers from Southern California Edison. 
http://www.sce.com/CustomerService/RateInformation/BusinessRates/LargeBusiness/

First off, you're looking at a 200kW load for 1000 nodes, which is a hefty 
load, just for the computers (not counting lights, HVAC, etc.)  But, no 
matter, we'll assume your facility is sucking at least 500kW some of the 
time, so that would put you in the large business TOU-8 tariff.
http://www.sce.com/NR/sc3/tm2/pdf/ce54-12.pdf
  All the large consumer tariffs are time-of-use sensitive.  I assume you 
wouldn't want some sort of "Critical Peak Pricing Options" or "Demand 
Bidding Programs"

Let's assume you're being served at 240V (as opposed to having your own 
distribution transformers, etc., although as a 200kW consumer, that's 
something you should consider).

Looks like the rates break down as about 0.016/kWh for the delivery, and 
the actual power (generation) runs somewhere between 0.04/kWh  (off peak 
summer) to 0.12/kWh on peak summer.

There's also a raft of other charges (metering, demand (runs about $10/kW 
of instantaneous demand), power factor, etc.)

Compare this to Domestic service.. where the rates run from 0.11 to 
0.18/kWh, depending on where you sit relative to baseline, season, etc. 
(I'll also point out that I was paying SCE $0.26/kWh at home in the summer 
of 2001, but rates are lower now.) Now, you might consider Residential to 
be artificially constrained for political reasons, so we take a look at 
GS-1 (general service..)
Here, we have 0.07 for delivery and 0.085 (totalling 0.155/kWh) during the 
summer.

The point is, there isn't a 10:1 ratio... not even close. And, rgb's 
ballpark of $0.10/kWh is a perfectly reasonable estimate, if a bit low for 
Southern California. Even as far back as 1990,on peak, large customers were 
paying on the order of $0.11/kWh.

It is possible that if you were buying power directly (which is possible, 
as a large end user), you can pay "market rate" for each kWh you 
consume.  The price is quite volatile, though... At the peak of the market 
failure a few years back, a kWh on the open market was something like $20 
during peak times.  I doubt you want to schedule your cluster ops to take 
advantage of electricity rate fluctuations.


>>Well, it isn't complete nonsense, unless you care to dispute the
>>number of days in a year, hours in a day, or cpus in a dual node
>>computer!
>>
>>The only term you're complaining about is the price of
>>electricity.  I'm not privy to the electrical rates that our
>>school pays, they may well be an order of magnitude lower.  My
>>home rates certainly aren't, but then, I don't buy as much
>>power as the campus.  It's also not at all clear that the
>>campus would sell power to the end users at the same rate
>>which it pays the utility.

CalTech probably buys their power from City of Pasadena, but it's probably 
similar in rate structure to SCE's TOU-8.  Home rates are somewhat 
artificially low for political reasons.  The folks really getting the short 
end of the stick are small businesses who don't have the negotiating power 
that a large business does, nor the political clout of elderly pensioners 
dying from heat.


I'll also note that if you start paying for kWh, you're going to want to 
give serious consideration to buying more nodes than you strictly need, and 
shutting down the cluster during on-peak times. A typical pricing strategy 
might be 0.15/0.07/0.05 (on/mid/off peak): Peak lasts 6 hrs (1200-1800), 
mid is 0800-1200,1800-2300, and offpeak is the rest.  There's 93 hours of 
offpeak time out of 168 total in the week

For the pricing and schedules above, it turns out that the optimum is to 
shut down only during peak, but run during midpeak, for an average energy 
cost of 0.056/kWh, but using 22% more computers.



James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875