[Beowulf] Academic sites: who pays for the electricity?
James.P.Lux at jpl.nasa.gov
Thu Feb 17 17:01:53 PST 2005
At 04:19 PM 2/17/2005, Jim Lux wrote:
>At 02:54 PM 2/17/2005, David Mathog wrote:
>>At Wed, 16 Feb 2005 19:08:05 +0100 Vincent Diepeveen wrote:
>> > Date: Wed, 16 Feb 2005 19:08:05 +0100
>> > From: Vincent Diepeveen <diep at xs4all.nl>
>> > Subject: Re: [Beowulf] Academic sites: who pays for the electricity?
>> > To: "David Mathog" <mathog at mendel.bio.caltech.edu>,
>> > beowulf at beowulf.org
>> > Message-ID: <188.8.131.5250216190804.0106fcc0 at pop.xs4all.nl>
>> > Content-Type: text/plain; charset="us-ascii"
>> > At 08:16 16-2-2005 -0800, David Mathog wrote:
>> > >In most universities services like electricity, water, and
>> > >A/C are paid for by the school. To do so they take "overhead"
>> > >out of every grant. Partially as a consequence of this they
>> > >typically have a very poor ability to meter usage on a room
>> > >by room basis.
>> > >
>> > >Now somewhere between the 10 node Pentium II beowulf sitting on
>> > >a lab bench and the 1000 node dual P4 Xeon beowulf in a machine
>> > >room that takes up half the basement the cost of the electricity
>> > >(both for power and A/C) goes from a minor expense to a major
>> > >one. Really major. For instance, in that hypothetical large machine,
>> > >at 10 cents per kilowatt hour (a round number), assuming 100 watts
>> > >per CPU (another round number) that's:
>> > >
>> > > 1000 (nodes) *
>> > > 2 (cpus/node) *
>> > > .1 (kilowatts/cpu) *
>> > > .1 (dollars/kilowatt-hour) *
>> > > 365 (days /year) *
>> > > 24 (hours/day) =
>> > >-----------------------
>> > > 175200 dollars/year
>> > Complete academic nonsense calculation. If you use quite some electricity
>> > the electricity gets up to factor 20-40 cheaper. Getting a factor 10
>> > reduction in usage bill is pretty easy if you negotiate properly.
Just where do you live that such negotiations are possible. Here's some
real numbers from Southern California Edison.
First off, you're looking at a 200kW load for 1000 nodes, which is a hefty
load, just for the computers (not counting lights, HVAC, etc.) But, no
matter, we'll assume your facility is sucking at least 500kW some of the
time, so that would put you in the large business TOU-8 tariff.
All the large consumer tariffs are time-of-use sensitive. I assume you
wouldn't want some sort of "Critical Peak Pricing Options" or "Demand
Let's assume you're being served at 240V (as opposed to having your own
distribution transformers, etc., although as a 200kW consumer, that's
something you should consider).
Looks like the rates break down as about 0.016/kWh for the delivery, and
the actual power (generation) runs somewhere between 0.04/kWh (off peak
summer) to 0.12/kWh on peak summer.
There's also a raft of other charges (metering, demand (runs about $10/kW
of instantaneous demand), power factor, etc.)
Compare this to Domestic service.. where the rates run from 0.11 to
0.18/kWh, depending on where you sit relative to baseline, season, etc.
(I'll also point out that I was paying SCE $0.26/kWh at home in the summer
of 2001, but rates are lower now.) Now, you might consider Residential to
be artificially constrained for political reasons, so we take a look at
GS-1 (general service..)
Here, we have 0.07 for delivery and 0.085 (totalling 0.155/kWh) during the
The point is, there isn't a 10:1 ratio... not even close. And, rgb's
ballpark of $0.10/kWh is a perfectly reasonable estimate, if a bit low for
Southern California. Even as far back as 1990,on peak, large customers were
paying on the order of $0.11/kWh.
It is possible that if you were buying power directly (which is possible,
as a large end user), you can pay "market rate" for each kWh you
consume. The price is quite volatile, though... At the peak of the market
failure a few years back, a kWh on the open market was something like $20
during peak times. I doubt you want to schedule your cluster ops to take
advantage of electricity rate fluctuations.
>>Well, it isn't complete nonsense, unless you care to dispute the
>>number of days in a year, hours in a day, or cpus in a dual node
>>The only term you're complaining about is the price of
>>electricity. I'm not privy to the electrical rates that our
>>school pays, they may well be an order of magnitude lower. My
>>home rates certainly aren't, but then, I don't buy as much
>>power as the campus. It's also not at all clear that the
>>campus would sell power to the end users at the same rate
>>which it pays the utility.
CalTech probably buys their power from City of Pasadena, but it's probably
similar in rate structure to SCE's TOU-8. Home rates are somewhat
artificially low for political reasons. The folks really getting the short
end of the stick are small businesses who don't have the negotiating power
that a large business does, nor the political clout of elderly pensioners
dying from heat.
I'll also note that if you start paying for kWh, you're going to want to
give serious consideration to buying more nodes than you strictly need, and
shutting down the cluster during on-peak times. A typical pricing strategy
might be 0.15/0.07/0.05 (on/mid/off peak): Peak lasts 6 hrs (1200-1800),
mid is 0800-1200,1800-2300, and offpeak is the rest. There's 93 hours of
offpeak time out of 168 total in the week
For the pricing and schedules above, it turns out that the optimum is to
shut down only during peak, but run during midpeak, for an average energy
cost of 0.056/kWh, but using 22% more computers.
James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
More information about the Beowulf