<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>One comment - my dissertation below is specifically about
non-ebullient immersion cooling. As Jim Lux pointed out in a later
e-mail, in ebullient cooling, some kind of surface feature to
promote nucleation could be beneficial. Ebbulient cooling is a
whole different beast from normal (non-ebullient) immersive
cooling, since in that case you have changes of state and gas
bubbles flowing through a liquid. <br>
</p>
<p>However, in all of the live and video demonstrations I've seen of
Novec, the processors were completely bare, bubbles were forming
at a pretty rapid rate, so again I think creating some sort of
heat sink for this would add cost with no significant benefit. <br>
</p>
<pre class="moz-signature" cols="72">Prentice Bisbal
Lead Software Engineer
Princeton Plasma Physics Laboratory
<a class="moz-txt-link-freetext" href="http://www.pppl.gov">http://www.pppl.gov</a></pre>
<div class="moz-cite-prefix">On 11/08/2018 10:40 AM, Prentice Bisbal
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:f1f082d1-e665-9a0c-2349-70ca229b6297@pppl.gov">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<p>Heat fins are used to increase the surface area used for heat
transfer, since the rate of energy transfer by conduction is
directly proportional the surface area. Heat fins are needed
when air is involved because air has such a low thermal
conductivity. <br>
</p>
<p>Thermal conductivity of liquids are much high, so heat fins
aren't as necessary. For example, I've read that water can
transfer heat orders of magnitude better than air, so using
water to remove hear from a processor would need orders of
magnitude less surface area for the same energy transfer rate. <br>
</p>
<p>Also, liquids have higher viscosities than gases, so we have to
worry about 'boundary layers'. A boundary layer is area where
the edge flowing fluid is in contact with a solid. The friction
between the liquid and the solid slows down the fluid near the
solid. This affects both gases and liquids, but since liquids
have higher viscosities, the effect is more noticeable. <br>
</p>
<p>Think about a car's radiator - the air side has all the fins on
it, and the liquid side has smooth pipe walls. <br>
</p>
<p><a class="moz-txt-link-freetext"
href="https://en.wikipedia.org/wiki/Boundary_layer"
moz-do-not-send="true">https://en.wikipedia.org/wiki/Boundary_layer</a></p>
<p>Convection heat transfer is an equally important mode of heat
transfer in fluids, and in the boundary layer, where the liquids
aren't moving as fast, heat transfer isn't as good, so you need
to keep your boundary layer from becoming too thick. <br>
</p>
<p>Since fluids have much higher thermal conductivities, and
boundary layer effects are more of a concern, I actually think a
smooth heat transfer surface would be better in these immersion
cooling cases. I'm sure smaller, more spaced out fins would
probably help heat transfer without creating too much of a
boundary layer, but making those heat sinks adds cost for
increased performance in a situation where it probably isn't
needed. <br>
</p>
<p>Now direct-contact cooling systems like Asetek products do
have fins on the liquid side, if I remember correctly, but that
in those systems, there are pumps to provide forced convection.
In immersion cooling, you are relying on natural convection, so
there isn't as much driving force to overcome viscosity/boundary
layer effects to force the liquid through the heat fins. <br>
</p>
<p>That's my thoughts, anyway. <br>
</p>
<pre class="moz-signature" cols="72">Prentice </pre>
<div class="moz-cite-prefix">On 11/07/2018 04:12 AM, John Hearns
via Beowulf wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAPqNE2XO_99NQ-3GP1ZixwfFoFzmFUGNMKErDHdRxHWB8e=AzQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<div dir="ltr">
<div dir="ltr">
<div>Thinking about liquid cooling , and the ebuillient
cooling, the main sources of heat on our current
architecture servers are the CPU package and the voltage
regulators. Then the DIMMs.</div>
<div>Concentrating on the CPU die package, it is engineered
with a flat metal surface which is intended to have a
thermal paste to transfer heat across to a flat metal
heatsink.</div>
<div>Those heatsinks are finned to have air blown across
them to transport the heat away.</div>
<div><br>
</div>
<div>In liquid immersion should we be looking at having a
spiky surface on the CPU die packages and the voltage
regulators?</div>
<div>Maybe we should spray the entire board with a
'flocking'' compound and give it a matt finish!</div>
<div>I am being semi-serious. I guess a lot of CFD
simulation done regarding air cooling with fins.</div>
<div>How much work has gone into pointy surfaces on the die
package, which would increase contact area of course and
also act as nucleation points for bubbles?</div>
<div><br>
</div>
<div>One interesting experiment to do - assuming the flat
areas of the CPU in an immersive system do not have (non
thermal paste) heatsinks bolted on:</div>
<div>take two systems and roughen up the die package
surfacewith sandpaper on one. Compare temperatures.</div>
<div><br>
</div>
<div>ps. I can't resist adding this. Sorry Stu . <a
href="https://www.youtube.com/watch?v=kHnifVTSFEo"
moz-do-not-send="true">https://www.youtube.com/watch?v=kHnifVTSFEo</a></div>
<div>I guess Kenneth Williams is a typical vendor Site
Engineer.</div>
<div>pps. the actress in the redress had her career ruined
by this film - she ver got a serious role again after
perfectly being typecast.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, 6 Nov 2018 at 22:57, Prentice Bisbal
via Beowulf <<a href="mailto:beowulf@beowulf.org"
moz-do-not-send="true">beowulf@beowulf.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_4669408001860479086moz-cite-prefix">On
11/06/2018 02:03 PM, Lux, Jim (337K) wrote:<br>
</div>
<blockquote type="cite">
<div class="m_4669408001860479086WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">True
enough.</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Ebullient
cooling does have some challenges – you can form
vapor films, which are good insulators, but if you
get the system working right, nothing beats phase
changes for a heat transfer. </span></p>
</div>
</blockquote>
If I recall what I learned in my Transport Phenomena
classes in engineering school, you need a reasonably high
temperature difference to get a stable film like that. For
that to happen, radiant heat transfer needs to be the
dominant heat transfer mechanism, in the range of
operation we are talking about, the temperature difference
isn't that great, and conduction is still the dominant
form of heat transfer. <br>
<br>
Here's an example of what 3M Novec ebullient cooling looks
like. It doesn't look like it's anywhere near the film
boiling regime: <br>
<br>
<a class="m_4669408001860479086moz-txt-link-freetext"
href="https://www.youtube.com/watch?v=CIbnl3Pj15w"
target="_blank" moz-do-not-send="true">https://www.youtube.com/watch?v=CIbnl3Pj15w</a><br>
<br>
--<br>
Prentice<br>
<br>
<blockquote type="cite">
<div class="m_4669408001860479086WordSection1"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"></span>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span></p>
<span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"></span>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">
Beowulf [<a
class="m_4669408001860479086moz-txt-link-freetext"
href="mailto:beowulf-bounces@beowulf.org"
target="_blank" moz-do-not-send="true">mailto:beowulf-bounces@beowulf.org</a>]
<b>On Behalf Of </b>Prentice Bisbal via
Beowulf<br>
<b>Sent:</b> Tuesday, November 06, 2018 8:17
AM<br>
<b>To:</b> <a
class="m_4669408001860479086moz-txt-link-abbreviated"
href="mailto:beowulf@beowulf.org"
target="_blank" moz-do-not-send="true">beowulf@beowulf.org</a><br>
<b>Subject:</b> Re: [Beowulf] More about those
underwater data centers</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">. And serviceability is
challenging. You need to pull the "wet" boards
out, or you need to connect and disconnect fluid
connectors, etc. If you're in an environment
where you can manage that (or are forced into it
by necessity), then you can do it.</p>
</blockquote>
<p>I think everyone on this list already knows I'm no
fan of mineral oil immersion (It just seems to messy
to me. Sorry, Stu), but immersion cooling with other
liquids, such as 3M Novec engineered fluid addresses
a lot of your concerns. It as a low boiling point,
not much above room temperature, and it was
originally meant to be an electronic parts cleaner
(according to a 3M rep at the 3M booth at SC a few
years ago, so if you pull a component out of it, it
dries very quickly and should be immaculately clean.
</p>
<p>The low boiling point is an excellent feature for
heat transfer, too, since it boils from the heat of
the processor (ebullient cooling). This change of
state absorbs a lot of energy, making it very
effective at transferring heat away from the
processor. The vapor can then rise and condense on a
heat exchanger with a chilled water heat exchanger,
where it again transfers a lot of heat through a
change of state. </p>
<pre>Prentice </pre>
<div>
<p class="MsoNormal">On 11/05/2018 06:30 PM, Stu
Midgley wrote:</p>
</div>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">I refute both these claims.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">You DO want to run your
boards immersed in coolant. It works
wonderfully well, is easy to live with,
servicing is easy... and saves you almost 1/2
your power bill.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">People are scared of
immersion cooling, but it isn't that difficult
to live with. Some things are harder but
other things are way easier. In total, it
balances out.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Also, given the greater
reliability of components you get, you do less
servicing.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">If you haven't lived with
it, you really have no idea what you are
missing.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Serviceability is NOT
challenging.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<p class="MsoNormal"> </p>
<div>
<blockquote style="border:none;border-left:solid
#cccccc 1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">You really do NOT want to
run boards immersed in coolant - yeah,
there's folks doing it at HPC scale<br>
<br>
Whatever the coolant, it leaks, it oozes, it
gets places you don't want it to go. And
serviceability is challenging. You need to
pull the "wet" boards out, or you need to
connect and disconnect fluid connectors,
etc. If you're in an environment where you
can manage that (or are forced into it by
necessity), then you can do it.</p>
</blockquote>
</div>
<p class="MsoNormal">-- </p>
<div>
<div>
<p class="MsoNormal">Dr Stuart Midgley<br>
<a href="mailto:sdm900@gmail.com"
target="_blank" moz-do-not-send="true">sdm900@gmail.com</a></p>
</div>
</div>
</div>
<p class="MsoNormal"><br>
<br>
<br>
</p>
<pre>_______________________________________________</pre>
<pre>Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored by Penguin Computing</pre>
<pre>To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" moz-do-not-send="true">http://www.beowulf.org/mailman/listinfo/beowulf</a></pre>
</blockquote>
<p class="MsoNormal"> </p>
</div>
</blockquote>
<br>
</div>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org"
target="_blank" moz-do-not-send="true">Beowulf@beowulf.org</a>
sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe)
visit <a
href="http://www.beowulf.org/mailman/listinfo/beowulf"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org" moz-do-not-send="true">Beowulf@beowulf.org</a> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit <a class="moz-txt-link-freetext" href="http://www.beowulf.org/mailman/listinfo/beowulf" moz-do-not-send="true">http://www.beowulf.org/mailman/listinfo/beowulf</a>
</pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>