[Beowulf] Servers Too Hot? Intel Recommends a Luxurious Oil Bath
Robert G. Brown
rgb at phy.duke.edu
Wed Sep 5 06:14:34 PDT 2012
On Tue, 4 Sep 2012, Ellis H. Wilson III wrote:
> Yes, Google does house these containers in a fairly basic building, but
> there is no reason I can think of why it couldn't put them out in the
> open and run all wires, etc, into the ground instead. I think they just
> put them in a building for convenience to the maintainers, rather than
> for some property of the building itself that would enable the
> containers to work better.
Google in particular, though, lives and dies by means of instantaneous
access to parts. A computer is to them as a mere neuron is to us --
nodes fail in their cluster at the rate of many a day, and are replaced
almost immediately the way they have things set up. This is multiply
economical for them -- minimum downtime, minimum human costs (because it
is EASY and FAST for them to pop a node out and a new one in), minimum
hardware costs because IIRC a "node" for them is literally a
motherboard, memory, CPU and it just fits into a harness in the
trailers, I don't think they even bother with a proper enclosure per
motherboard. Over the counter, commodity, cheap, almost hardware
agnostic.
Which are all reasons that it would be a terrible idea for Google to
fill the containers with any sort of gas or immerse the nodes in oil or
use any sort of non-contained direct-contact liquid to cool them. It
would take ten times as long to replace a node, literally. It would
mean (very probably) that they'd have to "mess" with the nodes in some
way putting them in -- I don't see normal CPU cooling fans moving oil,
for example, or there would be custom plumbing to a per-CPU, per-Mobo
water cooled sink that wouldn't work or would have to be replace if they
changed Mobo, or the fire/explosion risk and need to pump down an entire
container in order to replace a single motherboard, which might come
with a need to SHUT DOWN the entire container while this was going on.
Enormous expense.
Google is indeed the archetype for anyone seeking to build a truly
massive cluster for embarrassingly parallel applications that have
"value" only from the sheer volume of computations delivered. Screw
EVERYTHING fancy. Cheapest hardware but tons of it, rigged to be a five
minute replacement. I don't even think they try to repair the hardware
that fries -- if it fails for any reason they pull it, replace it, and
dump it. They can scale up indefinitely by just adding more trailers.
If you suggest adding oil or H2 or He4 to the trailers, they'll just
laugh. They don't need to overclock. They don't need bleeding edge
processors. If they need more capacity, they just add another trailer
full of processors all purchased square at the sweet spot for
price/performance.
rgb
>
> I still agree (see my email 3 back now) that there are major concerns
> with using hydrogen this way, but at massive scale and with massive
> precautions you might be able to extract some nice power savings with
> it. Maybe even enough to tolerate a few going sky high. Alas, my
> physical sciences skills are way to weak to get a bound on what those
> benefits might be.
>
> Best,
>
> ellis
>
>> And you ...
>> ... light up my server room
>> ... You make it hot
>>
>> And when sparks fly ...
>> ... so do my servers in
>> ... the ensuing explosion ...
>>
>> (ok, I'll keep the day job ... sheesh!)
>>
>>
>>> Ar 17.9
>>> CH4 34.1
>>>
>>> He also is hard to come by..
>>>
>>
>>
>>
>>>
>>> Jim Lux
>>>
>>> -----Original Message-----
>>> From: Robert G. Brown [mailto:rgb at phy.duke.edu]
>>> Sent: Tuesday, September 04, 2012 6:16 AM
>>> To: Lux, Jim (337C)
>>> Cc: Eugen Leitl; beowulf at beowulf.org
>>> Subject: Re: [Beowulf] Servers Too Hot? Intel Recommends a Luxurious Oil Bath
>>>
>>> On Mon, 3 Sep 2012, Lux, Jim (337C) wrote:
>>>
>>>> I'll bet they have to change it more often than that. This isnt
>>>> something like a pole transformer.
>>>
>>> Absolutely. Think of what you can do with a big vat of hot oil handy in the workspace. Buffalo Wings. French Fries. Chicken. Fish. The reason nobody does this is because OSHA prohibits it -- it is a huge health hazard. Not even Jolt Cola can keep you thin in a sedentary profession with your own personal deep frier as close as your server room. Although you do have to change the oil pretty often, as otherwise shrimp tails and bits of overcooked tempura crust gunk up the memory and CPU. Systems people were dying like pudgy little flies of advanced cardiovascular disease before the practice of using computers to heat deep fat was banned.
>>>
>>> On a more serious note, one wonders why nobody has tried helium instead.
>>> No, silly, not liquid helium, helium gas. The reason they fill windows with argon is that it has around 2/3 the thermal conductivity of air, and hence is a better insulator. This, in turn, is because it is more massive -- conductivity is tightly tied to mass and hence the speed of the molecules when they have kT sorts of energies.
>>>
>>> Helium, OTOH, has six times the thermal conductivity of air, and is relatively inexpensive. The biggest downside I can think of is that it requires a pretty good seal and thick walls to keep the slippery little atoms from sliding right through to the outside, and of course the fact that systems techs would always be hitting up the helium tanks so that they could talk like Donald Duck. And you'd still have to refrigerate the outside of the systems units. But all of these things are still orders of magnitude easier than with oil, and even things like cooling fans work fine in Helium. Maybe there are other problems -- lower heat capacity to match its higher conductivity -- but it seems like it is worth an experiment or two...
>>>
>>> rgb
>>>
>>> Robert G. Brown http://www.phy.duke.edu/~rgb/
>>> Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305
>>> Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu
>>>
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
Robert G. Brown http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu
More information about the Beowulf
mailing list