<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Stu, <br>
</p>
I have a BS in Chemical Engineering. Chemical engineers actually
spend more time studying heat, mass, and momentum transfer (aka<br>
heat transfer/thermodynamics, diffusion, and fluid mechanics), than
we do chemistry, so I think I have the proper training to evaluate
and comment on these various solutions. Just because I'm a system
administrator now, that shouldn't render my education null-and-void.
<br>
<br>
<br>
<div class="moz-cite-prefix">On 08/19/2016 10:59 AM, Stu Midgley
wrote:<br>
</div>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">Prentice, have you actually run any of these
solutions? Or are you just a back seat pundit?</div>
</blockquote>
No, I haven't run of these solutions,but that doesn't make the
science or practicality of these solutions any different. There are
a lot if astrophysicists who can tell you a lot about the Sun, Moon,
and stars without actually having travelled to any of them. <br>
<br>
If it makes you feel better, I typically spend hours at the
different cooling vendor booths each year talking about their
products because I can see how much better any form of liquid
cooling is than air-cooling is. <br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Immersion cooling is practical, efficient, easy and has a
ton of side benefits.</div>
</div>
</blockquote>
That can be said for all forms of liquid-cooling, not just mineral
oil immersion.<br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div> * cheap cooling (can run evaporative rather than
compressors)</div>
</div>
</blockquote>
Ditto. <br>
<br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> * nodes use 20% less energy due to removing fans</div>
</div>
</blockquote>
Ditto, again, In all cases, you still need pumps. Do the higher
viscosity of mineral oil, the pumps will need more power for the
same flow rate vs. water or Novec. If the thermal capacity is
higher, less flow rate would be needed, so that might be a wash. <br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> * node have less failures</div>
</div>
</blockquote>
Are you talking specifically about the reduction fan failures? If
so, that is certainly true, but it is true for all liquid cooling
methods** If your talking about other non-fan failures, I believe
those failures will be reduced by adequate cooling of the system
regardless of cooling method. <br>
<br>
** Non-immersive DCLC designs still need a fan on the power supply,
which could lead to fan failures and power supply failures so I'll
give you that. <br>
<br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> * system is more robust against cooling failures (ie. if
you lose cooling, tanks can take a long time to heat up to a
point where you would turn nodes off - we have ridden out
20mins cooling outages)</div>
</div>
</blockquote>
This is absolutely true. No arguments there. The small amounts of
water in a DCLC system have more capacity than air, but nothing
compared to a tank filled with mineral oil. <br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> * more reliable run times - cause cpu's never heat up and
change their performance</div>
</div>
</blockquote>
Again, this is independent of cooling mechanism. Adequate cooling
regardless of cooling mechanism will have the same effect. If you
want to make the argument that air is fickle when it comes to being
directed to the proper location in node (ie dead spots where there
is no airflow), I'll agree to that argument. You can get dead spots
with liquids, too, but because of their higher thermal conductivity,
they can still pull heat out of the 'dead zone' <br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> * fluid has a high fire and flash point - so you can get
permits to hold it</div>
</div>
</blockquote>
This isn't even a concern with water. I don't know the specifics of
3M Novec I tried to google it just now, but there's a a bunch of
different Novec fluids, and I don't know which one(s) are used for
immersion cooling, so I couldn't look up the safety information. I
don't have the time to dig any deeper. <br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> * fluid is readily available and cheap</div>
</div>
</blockquote>
So is water, and it's even cheaper! I assume the 3M Novec fluid also
readily available. Don't know the cost of it, though. <br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> * fluid is safe (I've ingested and swam in it with no
adverse effects - yet)</div>
</div>
</blockquote>
So is water, as long as you don't believe this guy:
<a class="moz-txt-link-freetext" href="http://www.dhmo.org/">http://www.dhmo.org/</a><br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> * quiet - no 1" screaming fans</div>
</div>
</blockquote>
Again, this applies to all liquid cooling solutions, but they still
need pumps somewhere. Okay the non-immersive DCLC solutions usually
don't cover the power supply, so you do still need fans for that. <br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I looked at all the other solutions and they scare the hell
out of me. Try putting all those pipes into a 2RU node with 8
phi's... it just won't happen.</div>
</div>
</blockquote>
I don't need to try to do that, because that's the job of the
engineers at Asetek, CoolIT and their OEM partners. I've looked at
their solutions every year they've been as SC, and the packaging is
quite neat, and included heat exchangers for the RAM to keep it
cool, too. I sure as hell wouldn't want to engineer and build the
system myself, but I'm sure glad there are others doing it, and I
have a lot of confidence in their work. <br>
<br>
Now my criticisms, which apply to mineral oil immersive cooling
only: <br>
<br>
* The stuff is messy. I can't imagine removing components from a
server without them slipping out of my hands. I can't even imagine
getting one from the tank to a workbench without getting oil
everywhere. I'm sure you can clean it off with a lot of electronic
parts cleaner, but how good for the environment is that? I see this
stuff gaining more market share when we get to 'disposable' servers
with ARM-based SoCs, so cleanup isn't even an issue. <br>
<br>
* Liquids are dense! Immersive cooling in mineral oil adds a lot to
the load of the data centers raised floor. Yes the tanks are about
the size of a rack flipped on it's side, but even the densest
servers are still mostly air by volume. I imagine many data centers
would need to upgrade their floors before going to this type of
cooling. <br>
<br>
* Since the "racks" are on their sides an only go up about 3 - 3.5
feet, it's not a very efficient use of space. We often talk about
data center space in square footage, but it's *volume* that really
matters, and I don't think mineral oil cooling is very space
efficient. <br>
<br>
* What if one of those tanks leaks or a pipe bursts? I'd rather
clean up water or Novec. Yes, there are electrical dangers when that
happens with water, but that's what circuit breakers and such (GFCI,
AFCI, etc.) are for. <br>
<br>
* What happens when you decommission hardware that's covered in
mineral oil? How do you clean it up? Will a recycling center take
electronics covered in mineral oil.<br>
<br>
--<br>
Prentice<br>
<br>
<br>
<blockquote
cite="mid:CAEM1RsXCDDHDnPqWJs=iow62+Tq+vTCsfwodmpfzX=HBPwSfuA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Aug 19, 2016 at 10:25 PM,
Prentice Bisbal <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:pbisbal@pppl.gov" target="_blank">pbisbal@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"><span
class="">On 08/19/2016 10:02 AM, Michael Di Domenico
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
On Fri, Aug 19, 2016 at 9:32 AM, Stu Midgley <<a
moz-do-not-send="true" href="mailto:sdm900@gmail.com"
target="_blank">sdm900@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<a moz-do-not-send="true"
href="http://www.intel.com.au/content/www/au/en/processors/xeon/xeon-e5-2600-v3-dugeo-insight-demo.html"
rel="noreferrer" target="_blank">http://www.intel.com.au/conten<wbr>t/www/au/en/processors/xeon/<wbr>xeon-e5-2600-v3-dugeo-insight-<wbr>demo.html</a><br>
<a moz-do-not-send="true"
href="http://insidehpc.com/2014/11/intel-xeon-phi-takes-downunder-geosolutions-new-depths/"
rel="noreferrer" target="_blank">http://insidehpc.com/2014/11/i<wbr>ntel-xeon-phi-takes-downunder-<wbr>geosolutions-new-depths/</a><br>
<a moz-do-not-send="true"
href="http://www.grcooling.com/team-member/downunder-geosolutions/"
rel="noreferrer" target="_blank">http://www.grcooling.com/team-<wbr>member/downunder-geosolutions/</a><br>
</blockquote>
thanks for the links....<br>
<br>
are you guys using oil immersed cooling? i've been
tracking it for<br>
years, but it still seems like it would make a big mess<br>
</blockquote>
</span>
I've ranted on the impracticalities of immersion cooling in
mineral oil on here many times in the past, so I won't
rehash those arguments again today. Suffice it to say I'm
pretty sure the guys at Green Revolution Cooling hate seeing
me at SC.<br>
<br>
I think direct contact liquid cooling (DCLC)solutions from
Asetek and CoolIT are much more practical, as are immersion
cooling systems using 3M Novec engineered fluid. Novec is
really appealing because of the heat transfer efficiency of
the change of state. I believe it was originally designed as
an electronic parts cleaner, too. It evaporates at room temp
like other electronic cleaners, so when you pull hardware
out the liquid, it's dry in a matter of seconds, and very
clean, too! Much more practical than mineral oil in that
regard. 3M claims it has a very low global warming factor,
too.<br>
<br>
Personally, I think DCLC since any leaks in the data center
should be smaller than if a tank-like chassis springs a
leak, and there one is scared of the environmental impact of
water, even when antifreeze and other water treatment
chemicals are added to it, as would be needed in this use
case.<br>
<br>
--<br>
Prentice
<div class="HOEnZb">
<div class="h5"><br>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a moz-do-not-send="true"
href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a>
sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe)
visit <a moz-do-not-send="true"
href="http://www.beowulf.org/mailman/listinfo/beowulf"
rel="noreferrer" target="_blank">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">Dr
Stuart Midgley<br>
<a moz-do-not-send="true" href="mailto:sdm900@sdm900.com"
target="_blank">sdm900@sdm900.com</a></div>
</div>
</blockquote>
<br>
</body>
</html>