[Beowulf] bring back 2012?

Prentice Bisbal pbisbal at pppl.gov
Fri Aug 19 09:39:34 PDT 2016


I have a BS in Chemical Engineering. Chemical engineers actually spend 
more time studying heat, mass, and momentum transfer (aka
heat transfer/thermodynamics, diffusion, and fluid mechanics), than we 
do chemistry, so I think I have the proper training to evaluate and 
comment on these various solutions. Just because I'm a system 
administrator now, that shouldn't render my education null-and-void.

On 08/19/2016 10:59 AM, Stu Midgley wrote:
> Prentice, have you actually run any of these solutions?  Or are you 
> just a back seat pundit?
No, I haven't run  of these solutions,but that doesn't make the science 
or practicality of these solutions any different. There are a lot if 
astrophysicists who can tell you a lot about the Sun, Moon, and stars 
without actually having travelled to any of them.

If it makes you feel better, I typically spend hours at the different 
cooling vendor booths each year talking about their products because I 
can see how much better any form of liquid cooling is than air-cooling is.
> Immersion cooling is practical, efficient, easy and has a ton of side 
> benefits.
That can be said for all forms of liquid-cooling, not just mineral oil 
>  * cheap cooling (can run evaporative rather than compressors)

>  * nodes use 20% less energy due to removing fans
Ditto, again, In all cases, you still need pumps. Do the higher 
viscosity of mineral oil, the pumps will need more power for the same 
flow rate vs. water or Novec. If the thermal capacity is higher, less 
flow rate would be needed, so that might be a wash.
>  * node have less failures
Are you talking specifically about the reduction fan failures? If so, 
that is certainly true, but it is true for all liquid cooling methods** 
If your talking about other non-fan failures, I believe those failures 
will be reduced by adequate cooling of the system regardless of cooling 

** Non-immersive DCLC designs still need a fan on the power supply, 
which could lead to fan failures and power supply failures so I'll give 
you that.

>  * system is more robust against cooling failures (ie. if you lose 
> cooling, tanks can take a long time to heat up to a point where you 
> would turn nodes off - we have ridden out 20mins cooling outages)
This is absolutely true. No arguments there. The small amounts of water 
in a DCLC system have more capacity than air, but nothing compared to a 
tank filled with mineral oil.
>  * more reliable run times - cause cpu's never heat up and change 
> their performance
Again, this is independent of cooling mechanism. Adequate cooling 
regardless of cooling mechanism will have the same effect. If you want 
to make the argument that air is fickle when it comes to being directed 
to the proper location in node (ie dead spots where there is no 
airflow), I'll agree to that argument. You can get dead spots with 
liquids, too, but because of their higher thermal conductivity, they can 
still pull heat out of the 'dead zone'
>  * fluid has a high fire and flash point - so you can get permits to 
> hold it
This isn't even a concern with water. I don't know the specifics of 3M 
Novec I tried to google it just now, but there's a a bunch of different 
Novec fluids, and I don't know which one(s) are used for immersion 
cooling, so I couldn't look up the safety information. I don't have the 
time to dig any deeper.
>  * fluid is readily available and cheap
So is water, and it's even cheaper! I assume the 3M Novec fluid also 
readily available. Don't know the cost of it, though.
>  * fluid is safe (I've ingested and swam in it with no adverse effects 
> - yet)
So is water, as long as you don't believe this guy: http://www.dhmo.org/
>  * quiet - no 1" screaming fans
Again, this applies to all liquid cooling solutions, but they still need 
pumps somewhere. Okay the non-immersive DCLC solutions usually don't 
cover the power supply, so you do still need fans for that.
> I looked at all the other solutions and they scare the hell out of 
> me.  Try putting all those pipes into a 2RU node with 8 phi's...  it 
> just won't happen.
I don't need to try to do that, because that's the job of the engineers 
at Asetek, CoolIT and their OEM partners. I've looked at their solutions 
every year they've been as SC, and the packaging is quite neat, and 
included heat exchangers for the RAM to keep it cool, too.  I sure as 
hell wouldn't want to engineer and build the system myself, but I'm sure 
glad there are others doing it, and I have a lot of confidence in their 

Now my criticisms, which apply to mineral oil immersive cooling only:

* The stuff is messy. I can't imagine removing components from a server 
without them slipping out of my hands. I can't even imagine getting one 
from the tank to a workbench without getting oil everywhere. I'm sure 
you can clean it off with a lot of electronic parts cleaner, but how 
good for the environment is that?  I see this stuff gaining more market 
share when we get to 'disposable' servers with ARM-based SoCs, so 
cleanup isn't even an issue.

* Liquids are dense! Immersive cooling in mineral oil adds a lot to the 
load of the data centers raised floor. Yes the tanks are about the size 
of a rack flipped on it's side, but even the densest servers are still 
mostly air by volume. I imagine many data centers would need to upgrade 
their floors before going to this type of cooling.

* Since the "racks" are on their sides an only go up about 3 - 3.5 feet, 
it's not a very efficient use of space. We often talk about data center 
space in square footage, but it's *volume* that really matters, and I 
don't think mineral oil cooling is very space efficient.

* What if one of those tanks leaks or a pipe bursts? I'd rather clean up 
water or Novec. Yes, there are electrical dangers when that happens with 
water, but that's what circuit breakers and such (GFCI, AFCI, etc.) are 

* What happens when you decommission hardware that's covered in mineral 
oil? How do you clean it up? Will a recycling center take electronics 
covered in mineral oil.


> On Fri, Aug 19, 2016 at 10:25 PM, Prentice Bisbal <pbisbal at pppl.gov 
> <mailto:pbisbal at pppl.gov>> wrote:
>     On 08/19/2016 10:02 AM, Michael Di Domenico wrote:
>         On Fri, Aug 19, 2016 at 9:32 AM, Stu Midgley <sdm900 at gmail.com
>         <mailto:sdm900 at gmail.com>> wrote:
>             http://www.intel.com.au/content/www/au/en/processors/xeon/xeon-e5-2600-v3-dugeo-insight-demo.html
>             <http://www.intel.com.au/content/www/au/en/processors/xeon/xeon-e5-2600-v3-dugeo-insight-demo.html>
>             http://insidehpc.com/2014/11/intel-xeon-phi-takes-downunder-geosolutions-new-depths/
>             <http://insidehpc.com/2014/11/intel-xeon-phi-takes-downunder-geosolutions-new-depths/>
>             http://www.grcooling.com/team-member/downunder-geosolutions/
>             <http://www.grcooling.com/team-member/downunder-geosolutions/>
>         thanks for the links....
>         are you guys using oil immersed cooling?  i've been tracking
>         it for
>         years, but it still seems like it would make a big mess
>     I've ranted on the impracticalities of immersion cooling in
>     mineral oil on here many times in the past, so I won't rehash
>     those arguments again today. Suffice it to say I'm pretty sure the
>     guys at Green Revolution Cooling hate seeing me at SC.
>     I think direct contact liquid cooling (DCLC)solutions from Asetek
>     and CoolIT are much more practical, as are immersion cooling
>     systems using 3M Novec engineered fluid. Novec is really appealing
>     because of the heat transfer efficiency of the change of state. I
>     believe it was originally designed as an electronic parts cleaner,
>     too. It evaporates at room temp like other electronic cleaners, so
>     when you pull hardware out the liquid, it's dry in a matter of
>     seconds, and very clean, too! Much more practical than mineral oil
>     in that regard. 3M claims it has a very low global warming factor,
>     too.
>     Personally, I think DCLC since any leaks in the data center should
>     be smaller than if a tank-like chassis springs a leak, and there
>     one is scared of the environmental impact of water, even when
>     antifreeze and other water treatment chemicals are added to it, as
>     would be needed in this use case.
>     --
>     Prentice
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>     <http://www.beowulf.org/mailman/listinfo/beowulf>
> -- 
> Dr Stuart Midgley
> sdm900 at sdm900.com <mailto:sdm900 at sdm900.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20160819/2bb1e172/attachment-0001.html>

More information about the Beowulf mailing list