<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I’ll bet the surface is rough enough that there are plenty of nucleation centers. Consider things like leads on parts.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Beowulf <beowulf-bounces@beowulf.org> on behalf of "beowulf@beowulf.org" <beowulf@beowulf.org><br>
<b>Reply-To: </b>Prentice Bisbal <pbisbal@pppl.gov><br>
<b>Date: </b>Thursday, November 8, 2018 at 7:47 AM<br>
<b>To: </b>"beowulf@beowulf.org" <beowulf@beowulf.org><br>
<b>Subject: </b>Re: [Beowulf] More about those underwater data centers<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p>One comment - my dissertation below is specifically about non-ebullient immersion cooling. As Jim Lux pointed out in a later e-mail, in ebullient cooling, some kind of surface feature to promote nucleation could be beneficial. Ebbulient cooling is a whole
different beast from normal (non-ebullient) immersive cooling, since in that case you have changes of state and gas bubbles flowing through a liquid.
<o:p></o:p></p>
<p>However, in all of the live and video demonstrations I've seen of Novec, the processors were completely bare, bubbles were forming at a pretty rapid rate, so again I think creating some sort of heat sink for this would add cost with no significant benefit.
<o:p></o:p></p>
<pre>Prentice Bisbal<o:p></o:p></pre>
<pre>Lead Software Engineer<o:p></o:p></pre>
<pre>Princeton Plasma Physics Laboratory<o:p></o:p></pre>
<pre><a href="http://www.pppl.gov">http://www.pppl.gov</a><o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/08/2018 10:40 AM, Prentice Bisbal wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p>Heat fins are used to increase the surface area used for heat transfer, since the rate of energy transfer by conduction is directly proportional the surface area. Heat fins are needed when air is involved because air has such a low thermal conductivity.
<o:p></o:p></p>
<p>Thermal conductivity of liquids are much high, so heat fins aren't as necessary. For example, I've read that water can transfer heat orders of magnitude better than air, so using water to remove hear from a processor would need orders of magnitude less surface
area for the same energy transfer rate. <o:p></o:p></p>
<p>Also, liquids have higher viscosities than gases, so we have to worry about 'boundary layers'. A boundary layer is area where the edge flowing fluid is in contact with a solid. The friction between the liquid and the solid slows down the fluid near the solid.
This affects both gases and liquids, but since liquids have higher viscosities, the effect is more noticeable.
<o:p></o:p></p>
<p>Think about a car's radiator - the air side has all the fins on it, and the liquid side has smooth pipe walls.
<o:p></o:p></p>
<p><a href="https://en.wikipedia.org/wiki/Boundary_layer">https://en.wikipedia.org/wiki/Boundary_layer</a><o:p></o:p></p>
<p>Convection heat transfer is an equally important mode of heat transfer in fluids, and in the boundary layer, where the liquids aren't moving as fast, heat transfer isn't as good, so you need to keep your boundary layer from becoming too thick.
<o:p></o:p></p>
<p>Since fluids have much higher thermal conductivities, and boundary layer effects are more of a concern, I actually think a smooth heat transfer surface would be better in these immersion cooling cases. I'm sure smaller, more spaced out fins would probably
help heat transfer without creating too much of a boundary layer, but making those heat sinks adds cost for increased performance in a situation where it probably isn't needed.
<o:p></o:p></p>
<p>Now direct-contact cooling systems like Asetek products do have fins on the liquid side, if I remember correctly, but that in those systems, there are pumps to provide forced convection. In immersion cooling, you are relying on natural convection, so there
isn't as much driving force to overcome viscosity/boundary layer effects to force the liquid through the heat fins.
<o:p></o:p></p>
<p>That's my thoughts, anyway. <o:p></o:p></p>
<pre>Prentice <o:p></o:p></pre>
<div>
<p class="MsoNormal">On 11/07/2018 04:12 AM, John Hearns via Beowulf wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">Thinking about liquid cooling , and the ebuillient cooling, the main sources of heat on our current architecture servers are the CPU package and the voltage regulators. Then the DIMMs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Concentrating on the CPU die package, it is engineered with a flat metal surface which is intended to have a thermal paste to transfer heat across to a flat metal heatsink.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Those heatsinks are finned to have air blown across them to transport the heat away.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In liquid immersion should we be looking at having a spiky surface on the CPU die packages and the voltage regulators?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Maybe we should spray the entire board with a 'flocking'' compound and give it a matt finish!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I am being semi-serious. I guess a lot of CFD simulation done regarding air cooling with fins.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">How much work has gone into pointy surfaces on the die package, which would increase contact area of course and also act as nucleation points for bubbles?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">One interesting experiment to do - assuming the flat areas of the CPU in an immersive system do not have (non thermal paste) heatsinks bolted on:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">take two systems and roughen up the die package surfacewith sandpaper on one. Compare temperatures.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">ps. I can't resist adding this. Sorry Stu . <a href="https://www.youtube.com/watch?v=kHnifVTSFEo">https://www.youtube.com/watch?v=kHnifVTSFEo</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I guess Kenneth Williams is a typical vendor Site Engineer.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">pps. the actress in the redress had her career ruined by this film - she ver got a serious role again after perfectly being typecast.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, 6 Nov 2018 at 22:57, Prentice Bisbal via Beowulf <<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">On 11/06/2018 02:03 PM, Lux, Jim (337K) wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">True enough.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D">Ebullient cooling does have some challenges – you can form vapor films, which are good insulators, but if you get the system working right, nothing beats
phase changes for a heat transfer. </span><o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal">If I recall what I learned in my Transport Phenomena classes in engineering school, you need a reasonably high temperature difference to get a stable film like that. For that to happen, radiant heat transfer needs to be the dominant heat
transfer mechanism, in the range of operation we are talking about, the temperature difference isn't that great, and conduction is still the dominant form of heat transfer.
<br>
<br>
Here's an example of what 3M Novec ebullient cooling looks like. It doesn't look like it's anywhere near the film boiling regime:
<br>
<br>
<a href="https://www.youtube.com/watch?v=CIbnl3Pj15w" target="_blank">https://www.youtube.com/watch?v=CIbnl3Pj15w</a><br>
<br>
--<br>
Prentice<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Beowulf [<a href="mailto:beowulf-bounces@beowulf.org" target="_blank">mailto:beowulf-bounces@beowulf.org</a>]
<b>On Behalf Of </b>Prentice Bisbal via Beowulf<br>
<b>Sent:</b> Tuesday, November 06, 2018 8:17 AM<br>
<b>To:</b> <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a><br>
<b>Subject:</b> Re: [Beowulf] More about those underwater data centers<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">. And serviceability is challenging. You need to pull the "wet" boards out, or you need to connect and disconnect fluid connectors, etc. If you're in an environment where you can
manage that (or are forced into it by necessity), then you can do it.<o:p></o:p></p>
</blockquote>
<p>I think everyone on this list already knows I'm no fan of mineral oil immersion (It just seems to messy to me. Sorry, Stu), but immersion cooling with other liquids, such as 3M Novec engineered fluid addresses a lot of your concerns. It as a low boiling
point, not much above room temperature, and it was originally meant to be an electronic parts cleaner (according to a 3M rep at the 3M booth at SC a few years ago, so if you pull a component out of it, it dries very quickly and should be immaculately clean.
<o:p></o:p></p>
<p>The low boiling point is an excellent feature for heat transfer, too, since it boils from the heat of the processor (ebullient cooling). This change of state absorbs a lot of energy, making it very effective at transferring heat away from the processor.
The vapor can then rise and condense on a heat exchanger with a chilled water heat exchanger, where it again transfers a lot of heat through a change of state.
<o:p></o:p></p>
<pre>Prentice <o:p></o:p></pre>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On 11/05/2018 06:30 PM, Stu Midgley wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I refute both these claims.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">You DO want to run your boards immersed in coolant. It works wonderfully well, is easy to live with, servicing is easy... and saves you almost 1/2 your power bill.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">People are scared of immersion cooling, but it isn't that difficult to live with. Some things are harder but other things are way easier. In total, it balances out.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Also, given the greater reliability of components you get, you do less servicing.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">If you haven't lived with it, you really have no idea what you are missing.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Serviceability is NOT challenging.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">You really do NOT want to run boards immersed in coolant - yeah, there's folks doing it at HPC scale<br>
<br>
Whatever the coolant, it leaks, it oozes, it gets places you don't want it to go. And serviceability is challenging. You need to pull the "wet" boards out, or you need to connect and disconnect fluid connectors, etc. If you're in an environment where you can
manage that (or are forced into it by necessity), then you can do it.<o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Dr Stuart Midgley<br>
<a href="mailto:sdm900@gmail.com" target="_blank">sdm900@gmail.com</a><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<o:p></o:p></pre>
<pre>To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">
http://www.beowulf.org/mailman/listinfo/beowulf</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing<o:p></o:p></pre>
<pre>To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf">http://www.beowulf.org/mailman/listinfo/beowulf</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
</div>
</body>
</html>