[Beowulf] power usage, Intel 5160 vs. AMD 2216

[Mark Hahn]

>> Vendor A estimates that at peak load a compute node with two AMD 2216s,
>> 4 GB of 667 DDR2 RAM, a hard drive, and an IB board will draw 265 watts.

that sounds right to me.  I have a cluster of dual-95W opteron nodes,
8GB ram, 2x HD, quadrics card, and they measure about that when under load.
(assuming it's not hyperventilating - when the fan controller gets upset,
dissipation for the 12 fans increases and draw is closer to 350.)

>> Vendor B estimates that such a node will draw 450 watts.

typical vendor response, based on PS rating.  it's only trivially correct,
but then again a mere sales droid can provide the answer ;)

> Considering that the regular 2216 is 95W peak (the 'HE' version is about
> 65W), and the memory and IB card are both pretty warm, 265 watts is
> unrealistic.

hmm.  I imagine there have been some warm IB cards, but most NICs I see
are in the ~8W range (quadrics, myrinet, IB, though _not_ counting those 
gross 10G TOE cards).

memory is typically about 1W/chip (.8W for the pc2-800 1Gb production chip 
from Micron I just looked at).  so 4x 1G 9-chip ECC dimms would total about
36W.

disks are only 5-15W.

> Multiply a realistic max power by a power supply
> efficiency and you'll get about 450 watts.

well, that seems a bit high to me, but not outlandish.  I'd probably
work with something between 350 and 400.

> There is way to get that lower number: 2216HE processors, and very efficient

HE model is $73 more according to AMD's current list price.  according to my
math (and local ~5 cent/kwh prices), it won't justify itself in direct 
operating cost, but might when you factor in lower infrastructure (wires,
cooling).

one interesting thing is that PSU's are most efficient near their max load.
so getting nodes with 500W PSUs (like mine), and then only running at 265W
is probably a bad thing.  I think I've read that a typical PSU might be 
only, say, 65% efficient at half-load, up to say 75 or 80% near full load.
so just sizing the PSU right could save almost the same power as HE CPUs...

> We've measured between 50% and 93% efficiency.  The worst are the supplies

Don, how do you measure efficiency?

> We could get into an interesting discussion about the best way to decrease
> the typical power use of a cluster.  The best way to do this is with

I've got disks in my cluster in standby most of the time - 
it'll only save a few watts per node, but the cost is negligable.

our cpus are, for better or worse, nearly 100% utilized all the time.
I don't know if there's any way to tell what fraction of time they are,
for instance, in blocking MPI communication.  for longer-term blocking
(perhaps local or network disk IO), I'd hope the CPU uses a high-savings
idle mode (halt, I guess).

hmm, I wonder if any nics can monitor how recently the host polled for
status, and use that to decide whether to issue an interrupt when a packet
comes in.  then the MPI lib could poll for the approximate RTT (say 5 us)
and sleep otherwise.  if the nic hadn't seen a poll for a few us, it could 
assume the host needed an irq to wake up...

then again, presumably 4-8 core chips will keep nics warmer...

regards, mark hahn.