[Beowulf] immersion cooling - Re: bring back 2012?

[Lux, Jim (337C)]

I have run oil cooled (and Fluorinert, similar to Novec, too) equipment.
Stu’s points are well taken, but the engineering challenges are substantial, if you don’t want a mess.  (capillary flow through wires, arghh)
And I can guarantee that your first few prototypes will make a mess, so learn how to clean up that spilled coolant

I think that from an engineering standpoint, dunking an assembly in a bucket is actually easier than trying to plumb little tiny spot coolers. The leak probability is proportional to the number of hoses and connections.

One early decision you’ll need to make is about replaceability and serviceability.  If you can just over provision and leave dead nodes in the tank, then this probably isn’t as much an issue.  These days, “swapping parts” is something we just don’t do. Nobody upgrades a processor on an existing mobo any more.

There’s also the regulatory/architectural/infrastructure issues – it’s one thing to have a small tank full of a few gallons of oil stashed in your lab, another to have a data center with 1000 nodes with thousands of gallons.  That scale is going to require things like special containment barriers in the event of a leak(you’ve probably seen the concrete curb or berm around outdoor storage tanks), signage for emergency responders, procedures for emergency response, etc.  None of this is difficult, but it’s “out of the ordinary”, and so, will require more work than “install a standard data center”

Novec and similar engineered fluids are pretty expensive compared to oil. Last time I priced mineral oil in large lots (1000 gallons) it was in the $2-3/gallon range (basically, twice the per bbl price of crude oil).


From: Beowulf <beowulf-bounces at beowulf.org<mailto:beowulf-bounces at beowulf.org>> on behalf of Stu Midgley <sdm900 at gmail.com<mailto:sdm900 at gmail.com>>
Reply-To: "sdm900 at gmail.com<mailto:sdm900 at gmail.com>" <sdm900 at gmail.com<mailto:sdm900 at gmail.com>>
Date: Friday, August 19, 2016 at 7:59 AM
To: Prentice Bisbal <pbisbal at pppl.gov<mailto:pbisbal at pppl.gov>>
Cc: "beowulf at beowulf.org<mailto:beowulf at beowulf.org>" <beowulf at beowulf.org<mailto:beowulf at beowulf.org>>
Subject: Re: [Beowulf] bring back 2012?

Prentice, have you actually run any of these solutions?  Or are you just a back seat pundit?

Immersion cooling is practical, efficient, easy and has a ton of side benefits.

 * cheap cooling (can run evaporative rather than compressors)
 * nodes use 20% less energy due to removing fans
 * node have less failures
 * system is more robust against cooling failures (ie. if you lose cooling, tanks can take a long time to heat up to a point where you would turn nodes off - we have ridden out 20mins cooling outages)
 * more reliable run times - cause cpu's never heat up and change their performance
 * fluid has a high fire and flash point - so you can get permits to hold it
 * fluid is readily available and cheap
 * fluid is safe (I've ingested and swam in it with no adverse effects - yet)
 * quiet - no 1" screaming fans

I looked at all the other solutions and they scare the hell out of me.  Try putting all those pipes into a 2RU node with 8 phi's...  it just won't happen.


On Fri, Aug 19, 2016 at 10:25 PM, Prentice Bisbal <pbisbal at pppl.gov<mailto:pbisbal at pppl.gov>> wrote:
On 08/19/2016 10:02 AM, Michael Di Domenico wrote:

On Fri, Aug 19, 2016 at 9:32 AM, Stu Midgley <sdm900 at gmail.com<mailto:sdm900 at gmail.com>> wrote:
http://www.intel.com.au/content/www/au/en/processors/xeon/xeon-e5-2600-v3-dugeo-insight-demo.html
http://insidehpc.com/2014/11/intel-xeon-phi-takes-downunder-geosolutions-new-depths/
http://www.grcooling.com/team-member/downunder-geosolutions/
thanks for the links....

are you guys using oil immersed cooling?  i've been tracking it for
years, but it still seems like it would make a big mess
I've ranted on the impracticalities of immersion cooling in mineral oil on here many times in the past, so I won't rehash those arguments again today. Suffice it to say I'm pretty sure the guys at Green Revolution Cooling hate seeing me at SC.

I think direct contact liquid cooling (DCLC)solutions from Asetek and CoolIT are much more practical, as are immersion cooling systems using 3M Novec engineered fluid. Novec is really appealing because of the heat transfer efficiency of the change of state. I believe it was originally designed as an electronic parts cleaner, too. It evaporates at room temp like other electronic cleaners, so when you pull hardware out the liquid, it's dry in a matter of seconds, and very clean, too! Much more practical than mineral oil in that regard. 3M claims it has a very low global warming factor, too.

Personally, I think DCLC since any leaks in the data center should be smaller than if a tank-like chassis springs a leak, and there one is scared of the environmental impact of water, even when antifreeze and other water treatment chemicals are added to it, as would be needed in this use case.

--
Prentice

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



--
Dr Stuart Midgley
sdm900 at sdm900.com<mailto:sdm900 at sdm900.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20160907/b31452bf/attachment.html>