[Beowulf] bring back 2012?
Prentice Bisbal
pbisbal at pppl.gov
Fri Aug 19 09:39:34 PDT 2016
Stu,
I have a BS in Chemical Engineering. Chemical engineers actually spend
more time studying heat, mass, and momentum transfer (aka
heat transfer/thermodynamics, diffusion, and fluid mechanics), than we
do chemistry, so I think I have the proper training to evaluate and
comment on these various solutions. Just because I'm a system
administrator now, that shouldn't render my education null-and-void.
On 08/19/2016 10:59 AM, Stu Midgley wrote:
> Prentice, have you actually run any of these solutions? Or are you
> just a back seat pundit?
No, I haven't run of these solutions,but that doesn't make the science
or practicality of these solutions any different. There are a lot if
astrophysicists who can tell you a lot about the Sun, Moon, and stars
without actually having travelled to any of them.
If it makes you feel better, I typically spend hours at the different
cooling vendor booths each year talking about their products because I
can see how much better any form of liquid cooling is than air-cooling is.
>
> Immersion cooling is practical, efficient, easy and has a ton of side
> benefits.
That can be said for all forms of liquid-cooling, not just mineral oil
immersion.
>
> * cheap cooling (can run evaporative rather than compressors)
Ditto.
> * nodes use 20% less energy due to removing fans
Ditto, again, In all cases, you still need pumps. Do the higher
viscosity of mineral oil, the pumps will need more power for the same
flow rate vs. water or Novec. If the thermal capacity is higher, less
flow rate would be needed, so that might be a wash.
> * node have less failures
Are you talking specifically about the reduction fan failures? If so,
that is certainly true, but it is true for all liquid cooling methods**
If your talking about other non-fan failures, I believe those failures
will be reduced by adequate cooling of the system regardless of cooling
method.
** Non-immersive DCLC designs still need a fan on the power supply,
which could lead to fan failures and power supply failures so I'll give
you that.
> * system is more robust against cooling failures (ie. if you lose
> cooling, tanks can take a long time to heat up to a point where you
> would turn nodes off - we have ridden out 20mins cooling outages)
This is absolutely true. No arguments there. The small amounts of water
in a DCLC system have more capacity than air, but nothing compared to a
tank filled with mineral oil.
> * more reliable run times - cause cpu's never heat up and change
> their performance
Again, this is independent of cooling mechanism. Adequate cooling
regardless of cooling mechanism will have the same effect. If you want
to make the argument that air is fickle when it comes to being directed
to the proper location in node (ie dead spots where there is no
airflow), I'll agree to that argument. You can get dead spots with
liquids, too, but because of their higher thermal conductivity, they can
still pull heat out of the 'dead zone'
> * fluid has a high fire and flash point - so you can get permits to
> hold it
This isn't even a concern with water. I don't know the specifics of 3M
Novec I tried to google it just now, but there's a a bunch of different
Novec fluids, and I don't know which one(s) are used for immersion
cooling, so I couldn't look up the safety information. I don't have the
time to dig any deeper.
> * fluid is readily available and cheap
So is water, and it's even cheaper! I assume the 3M Novec fluid also
readily available. Don't know the cost of it, though.
> * fluid is safe (I've ingested and swam in it with no adverse effects
> - yet)
So is water, as long as you don't believe this guy: http://www.dhmo.org/
> * quiet - no 1" screaming fans
Again, this applies to all liquid cooling solutions, but they still need
pumps somewhere. Okay the non-immersive DCLC solutions usually don't
cover the power supply, so you do still need fans for that.
>
> I looked at all the other solutions and they scare the hell out of
> me. Try putting all those pipes into a 2RU node with 8 phi's... it
> just won't happen.
I don't need to try to do that, because that's the job of the engineers
at Asetek, CoolIT and their OEM partners. I've looked at their solutions
every year they've been as SC, and the packaging is quite neat, and
included heat exchangers for the RAM to keep it cool, too. I sure as
hell wouldn't want to engineer and build the system myself, but I'm sure
glad there are others doing it, and I have a lot of confidence in their
work.
Now my criticisms, which apply to mineral oil immersive cooling only:
* The stuff is messy. I can't imagine removing components from a server
without them slipping out of my hands. I can't even imagine getting one
from the tank to a workbench without getting oil everywhere. I'm sure
you can clean it off with a lot of electronic parts cleaner, but how
good for the environment is that? I see this stuff gaining more market
share when we get to 'disposable' servers with ARM-based SoCs, so
cleanup isn't even an issue.
* Liquids are dense! Immersive cooling in mineral oil adds a lot to the
load of the data centers raised floor. Yes the tanks are about the size
of a rack flipped on it's side, but even the densest servers are still
mostly air by volume. I imagine many data centers would need to upgrade
their floors before going to this type of cooling.
* Since the "racks" are on their sides an only go up about 3 - 3.5 feet,
it's not a very efficient use of space. We often talk about data center
space in square footage, but it's *volume* that really matters, and I
don't think mineral oil cooling is very space efficient.
* What if one of those tanks leaks or a pipe bursts? I'd rather clean up
water or Novec. Yes, there are electrical dangers when that happens with
water, but that's what circuit breakers and such (GFCI, AFCI, etc.) are
for.
* What happens when you decommission hardware that's covered in mineral
oil? How do you clean it up? Will a recycling center take electronics
covered in mineral oil.
--
Prentice
>
>
> On Fri, Aug 19, 2016 at 10:25 PM, Prentice Bisbal <pbisbal at pppl.gov
> <mailto:pbisbal at pppl.gov>> wrote:
>
> On 08/19/2016 10:02 AM, Michael Di Domenico wrote:
>
> On Fri, Aug 19, 2016 at 9:32 AM, Stu Midgley <sdm900 at gmail.com
> <mailto:sdm900 at gmail.com>> wrote:
>
> http://www.intel.com.au/content/www/au/en/processors/xeon/xeon-e5-2600-v3-dugeo-insight-demo.html
> <http://www.intel.com.au/content/www/au/en/processors/xeon/xeon-e5-2600-v3-dugeo-insight-demo.html>
> http://insidehpc.com/2014/11/intel-xeon-phi-takes-downunder-geosolutions-new-depths/
> <http://insidehpc.com/2014/11/intel-xeon-phi-takes-downunder-geosolutions-new-depths/>
> http://www.grcooling.com/team-member/downunder-geosolutions/
> <http://www.grcooling.com/team-member/downunder-geosolutions/>
>
> thanks for the links....
>
> are you guys using oil immersed cooling? i've been tracking
> it for
> years, but it still seems like it would make a big mess
>
> I've ranted on the impracticalities of immersion cooling in
> mineral oil on here many times in the past, so I won't rehash
> those arguments again today. Suffice it to say I'm pretty sure the
> guys at Green Revolution Cooling hate seeing me at SC.
>
> I think direct contact liquid cooling (DCLC)solutions from Asetek
> and CoolIT are much more practical, as are immersion cooling
> systems using 3M Novec engineered fluid. Novec is really appealing
> because of the heat transfer efficiency of the change of state. I
> believe it was originally designed as an electronic parts cleaner,
> too. It evaporates at room temp like other electronic cleaners, so
> when you pull hardware out the liquid, it's dry in a matter of
> seconds, and very clean, too! Much more practical than mineral oil
> in that regard. 3M claims it has a very low global warming factor,
> too.
>
> Personally, I think DCLC since any leaks in the data center should
> be smaller than if a tank-like chassis springs a leak, and there
> one is scared of the environmental impact of water, even when
> antifreeze and other water treatment chemicals are added to it, as
> would be needed in this use case.
>
> --
> Prentice
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> <http://www.beowulf.org/mailman/listinfo/beowulf>
>
>
>
>
> --
> Dr Stuart Midgley
> sdm900 at sdm900.com <mailto:sdm900 at sdm900.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20160819/2bb1e172/attachment-0001.html>
More information about the Beowulf
mailing list