<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>I have run oil cooled (and Fluorinert, similar to Novec, too) equipment.</div>
</div>
</div>
<div>Stu’s points are well taken, but the engineering challenges are substantial, if you don’t want a mess. (capillary flow through wires, arghh)</div>
<div>And I can guarantee that your first few prototypes will make a mess, so learn how to clean up that spilled coolant</div>
<div><br>
</div>
<div>I think that from an engineering standpoint, dunking an assembly in a bucket is actually easier than trying to plumb little tiny spot coolers. The leak probability is proportional to the number of hoses and connections.</div>
<div><br>
</div>
<div>One early decision you’ll need to make is about replaceability and serviceability. If you can just over provision and leave dead nodes in the tank, then this probably isn’t as much an issue. These days, “swapping parts” is something we just don’t do.
Nobody upgrades a processor on an existing mobo any more.</div>
<div><br>
</div>
<div>There’s also the regulatory/architectural/infrastructure issues – it’s one thing to have a small tank full of a few gallons of oil stashed in your lab, another to have a data center with 1000 nodes with thousands of gallons. That scale is going to require
things like special containment barriers in the event of a leak(you’ve probably seen the concrete curb or berm around outdoor storage tanks), signage for emergency responders, procedures for emergency response, etc. None of this is difficult, but it’s “out
of the ordinary”, and so, will require more work than “install a standard data center”</div>
<div><br>
</div>
<div>Novec and similar engineered fluids are pretty expensive compared to oil. Last time I priced mineral oil in large lots (1000 gallons) it was in the $2-3/gallon range (basically, twice the per bbl price of crude oil).</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Beowulf <<a href="mailto:beowulf-bounces@beowulf.org">beowulf-bounces@beowulf.org</a>> on behalf of Stu Midgley <<a href="mailto:sdm900@gmail.com">sdm900@gmail.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"<a href="mailto:sdm900@gmail.com">sdm900@gmail.com</a>" <<a href="mailto:sdm900@gmail.com">sdm900@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, August 19, 2016 at 7:59 AM<br>
<span style="font-weight:bold">To: </span>Prentice Bisbal <<a href="mailto:pbisbal@pppl.gov">pbisbal@pppl.gov</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>" <<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Beowulf] bring back 2012?<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Prentice, have you actually run any of these solutions? Or are you just a back seat pundit?
<div><br>
</div>
<div>Immersion cooling is practical, efficient, easy and has a ton of side benefits.</div>
<div><br>
</div>
<div> * cheap cooling (can run evaporative rather than compressors)</div>
<div> * nodes use 20% less energy due to removing fans</div>
<div> * node have less failures</div>
<div> * system is more robust against cooling failures (ie. if you lose cooling, tanks can take a long time to heat up to a point where you would turn nodes off - we have ridden out 20mins cooling outages)</div>
<div> * more reliable run times - cause cpu's never heat up and change their performance</div>
<div> * fluid has a high fire and flash point - so you can get permits to hold it</div>
<div> * fluid is readily available and cheap</div>
<div> * fluid is safe (I've ingested and swam in it with no adverse effects - yet)</div>
<div> * quiet - no 1" screaming fans</div>
<div><br>
</div>
<div>I looked at all the other solutions and they scare the hell out of me. Try putting all those pipes into a 2RU node with 8 phi's... it just won't happen.</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Aug 19, 2016 at 10:25 PM, Prentice Bisbal <span dir="ltr">
<<a href="mailto:pbisbal@pppl.gov" target="_blank">pbisbal@pppl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="">On 08/19/2016 10:02 AM, Michael Di Domenico wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Fri, Aug 19, 2016 at 9:32 AM, Stu Midgley <<a href="mailto:sdm900@gmail.com" target="_blank">sdm900@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<a href="http://www.intel.com.au/content/www/au/en/processors/xeon/xeon-e5-2600-v3-dugeo-insight-demo.html" rel="noreferrer" target="_blank">http://www.intel.com.au/conten<wbr>t/www/au/en/processors/xeon/<wbr>xeon-e5-2600-v3-dugeo-insight-<wbr>demo.html</a><br>
<a href="http://insidehpc.com/2014/11/intel-xeon-phi-takes-downunder-geosolutions-new-depths/" rel="noreferrer" target="_blank">http://insidehpc.com/2014/11/i<wbr>ntel-xeon-phi-takes-downunder-<wbr>geosolutions-new-depths/</a><br>
<a href="http://www.grcooling.com/team-member/downunder-geosolutions/" rel="noreferrer" target="_blank">http://www.grcooling.com/team-<wbr>member/downunder-geosolutions/</a><br>
</blockquote>
thanks for the links....<br>
<br>
are you guys using oil immersed cooling? i've been tracking it for<br>
years, but it still seems like it would make a big mess<br>
</blockquote>
</span>I've ranted on the impracticalities of immersion cooling in mineral oil on here many times in the past, so I won't rehash those arguments again today. Suffice it to say I'm pretty sure the guys at Green Revolution Cooling hate seeing me at SC.<br>
<br>
I think direct contact liquid cooling (DCLC)solutions from Asetek and CoolIT are much more practical, as are immersion cooling systems using 3M Novec engineered fluid. Novec is really appealing because of the heat transfer efficiency of the change of state.
I believe it was originally designed as an electronic parts cleaner, too. It evaporates at room temp like other electronic cleaners, so when you pull hardware out the liquid, it's dry in a matter of seconds, and very clean, too! Much more practical than mineral
oil in that regard. 3M claims it has a very low global warming factor, too.<br>
<br>
Personally, I think DCLC since any leaks in the data center should be smaller than if a tank-like chassis springs a leak, and there one is scared of the environmental impact of water, even when antifreeze and other water treatment chemicals are added to it,
as would be needed in this use case.<br>
<br>
--<br>
Prentice
<div class="HOEnZb">
<div class="h5"><br>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">
http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">Dr Stuart Midgley<br>
<a href="mailto:sdm900@sdm900.com" target="_blank">sdm900@sdm900.com</a></div>
</div>
</div>
</div>
</span>
</body>
</html>