[Beowulf] More about those underwater data centers

Tue Nov 13 08:31:03 PST 2018

Ooooh…. liquid cooling video from sC18
https://twitter.com/Yuryu/status/1062178413270786048

On Fri, 9 Nov 2018 at 03:42, Lux, Jim (337K) via Beowulf <
beowulf at beowulf.org> wrote:

> I’ll bet the surface is rough enough that there are plenty of nucleation
> centers.  Consider things like leads on parts.
>
>
>
> *From: *Beowulf <beowulf-bounces at beowulf.org> on behalf of "
> beowulf at beowulf.org" <beowulf at beowulf.org>
> *Reply-To: *Prentice Bisbal <pbisbal at pppl.gov>
> *Date: *Thursday, November 8, 2018 at 7:47 AM
> *To: *"beowulf at beowulf.org" <beowulf at beowulf.org>
> *Subject: *Re: [Beowulf] More about those underwater data centers
>
>
>
> One comment - my dissertation below is specifically about non-ebullient
> immersion cooling. As Jim Lux pointed out in a later e-mail, in ebullient
> cooling, some kind of surface feature to promote nucleation could be
> beneficial. Ebbulient cooling is a whole different beast from normal
> (non-ebullient) immersive cooling, since in that case you have changes of
> state and gas bubbles flowing through a liquid.
>
> However, in all of the live and video demonstrations I've seen of Novec,
> the processors were completely bare, bubbles were forming at a pretty rapid
> rate, so again I think creating some sort of heat sink for this would add
> cost with no significant benefit.
>
> Prentice Bisbal
>
> Lead Software Engineer
>
> Princeton Plasma Physics Laboratory
>
> http://www.pppl.gov
>
> On 11/08/2018 10:40 AM, Prentice Bisbal wrote:
>
> Heat fins are used to increase the surface area used for heat transfer,
> since the rate of energy transfer by conduction is directly proportional
> the surface area. Heat fins are needed when air is involved because air has
> such a low thermal conductivity.
>
> Thermal conductivity of liquids are much high, so heat fins aren't as
> necessary. For example, I've read that water can transfer heat orders of
> magnitude better than air, so using water to remove hear from a processor
> would need orders of magnitude less surface area for the same energy
> transfer rate.
>
> Also, liquids have higher viscosities than gases, so we have to worry
> about 'boundary layers'. A boundary layer is area where the edge flowing
> fluid is in contact with a solid. The friction between the liquid and the
> solid slows down the fluid near the solid. This affects both gases and
> liquids, but since liquids have higher viscosities, the effect is more
> noticeable.
>
> Think about a car's radiator - the air side has all the fins on it, and
> the liquid side has smooth pipe walls.
>
> https://en.wikipedia.org/wiki/Boundary_layer
>
> Convection heat transfer is an equally important mode of heat transfer in
> fluids, and in the boundary layer, where the liquids aren't moving as fast,
> heat transfer isn't as good, so you need to keep your boundary layer from
> becoming too thick.
>
> Since fluids have much higher thermal conductivities, and boundary layer
> effects are more of a concern, I actually think a smooth heat transfer
> surface would be better in these immersion cooling cases. I'm sure
> smaller,  more spaced out fins would probably help heat transfer without
> creating too much of a boundary layer, but making those heat sinks adds
> cost for increased performance in a situation where it probably isn't
> needed.
>
> Now direct-contact cooling systems like Asetek products  do have fins on
> the liquid side, if I remember correctly, but that in those systems, there
> are pumps to provide forced convection. In immersion cooling, you are
> relying on natural convection, so there isn't as much driving force to
> overcome viscosity/boundary layer effects to force the liquid through the
> heat fins.
>
> That's my thoughts, anyway.
>
> Prentice
>
> On 11/07/2018 04:12 AM, John Hearns via Beowulf wrote:
>
> Thinking about liquid cooling , and the ebuillient cooling, the main
> sources of heat on our current architecture servers are the CPU package and
> the voltage regulators. Then the DIMMs.
>
> Concentrating on the CPU die package, it is engineered with a flat metal
> surface which is intended to have a thermal paste to transfer heat across
> to a flat metal heatsink.
>
> Those heatsinks are finned to have air blown across them to transport the
> heat away.
>
>
>
> In liquid immersion should we be looking at having a spiky surface on the
> CPU die packages and the voltage regulators?
>
> Maybe we should spray the entire board with a 'flocking'' compound and
> give it a matt finish!
>
> I am being semi-serious. I guess a lot of CFD simulation  done regarding
> air cooling with fins.
>
> How much work has gone into pointy surfaces on the die package, which
> would increase contact area of course and also act as nucleation points for
> bubbles?
>
>
>
> One interesting experiment to do - assuming the flat areas of the CPU in
> an immersive system do not have (non thermal paste) heatsinks bolted on:
>
> take two systems and roughen up the die package surfacewith sandpaper on
> one. Compare temperatures.
>
>
>
> ps. I can't resist adding this. Sorry Stu .
> https://www.youtube.com/watch?v=kHnifVTSFEo
>
> I guess Kenneth Williams is a typical vendor Site Engineer.
>
> pps. the actress in the redress had her career ruined by this film - she
> ver got a serious role again after perfectly being typecast.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, 6 Nov 2018 at 22:57, Prentice Bisbal via Beowulf <
> beowulf at beowulf.org> wrote:
>
> On 11/06/2018 02:03 PM, Lux, Jim (337K) wrote:
>
> True enough.
>
> Ebullient cooling does have some challenges – you can form vapor films,
> which are good insulators, but if you get the system working right, nothing
> beats phase changes for a heat transfer.
>
> If I recall what I learned in my Transport Phenomena classes in
> engineering school, you need a reasonably high temperature difference to
> get a stable film like that. For that to happen, radiant heat transfer
> needs to be the dominant heat transfer mechanism, in the range of operation
> we are talking about, the temperature difference isn't that great, and
> conduction is still the dominant form of heat transfer.
>
> Here's an example of what 3M Novec ebullient cooling looks like. It
> doesn't look like it's anywhere near the film boiling regime:
>
> https://www.youtube.com/watch?v=CIbnl3Pj15w
>
> --
> Prentice
>
>
>
>
>
>
> *From:* Beowulf [mailto:beowulf-bounces at beowulf.org
> <beowulf-bounces at beowulf.org>] *On Behalf Of *Prentice Bisbal via Beowulf
> *Sent:* Tuesday, November 06, 2018 8:17 AM
> *To:* beowulf at beowulf.org
> *Subject:* Re: [Beowulf] More about those underwater data centers
>
>
>
> . And serviceability is challenging. You need to pull the "wet" boards
> out, or you need to connect and disconnect fluid connectors, etc.  If
> you're in an environment where you can manage that (or are forced into it
> by necessity), then you can do it.
>
> I think everyone on this list already knows I'm no fan of mineral oil
> immersion (It just seems to messy to me. Sorry, Stu), but immersion cooling
> with other liquids, such as 3M Novec engineered fluid addresses a lot of
> your concerns. It as a low boiling point, not much above room temperature,
> and it was originally meant to be an electronic parts cleaner (according to
> a 3M rep at the 3M booth at SC a few years ago, so if you pull a component
> out of it, it dries very quickly and should be immaculately clean.
>
> The low boiling point is an excellent feature for heat transfer, too,
> since it boils from the heat of the processor (ebullient cooling). This
> change of state absorbs a lot of energy, making it very effective at
> transferring heat away from the processor. The vapor can then rise and
> condense on a heat exchanger with a chilled water heat exchanger, where it
> again transfers a lot of heat through a change of state.
>
> Prentice
>
> On 11/05/2018 06:30 PM, Stu Midgley wrote:
>
> I refute both these claims.
>
>
>
> You DO want to run your boards immersed in coolant.  It works wonderfully
> well, is easy to live with, servicing is easy... and saves you almost 1/2
> your power bill.
>
>
>
> People are scared of immersion cooling, but it isn't that difficult to
> live with.  Some things are harder but other things are way easier.  In
> total, it balances out.
>
>
>
> Also, given the greater reliability of components you get, you do less
> servicing.
>
>
>
> If you haven't lived with it, you really have no idea what you are missing.
>
>
>
>
>
> Serviceability is NOT challenging.
>
>
>
>
>
>
>
> You really do NOT want to run boards immersed in coolant - yeah, there's
> folks doing it at HPC scale
>
> Whatever the coolant, it leaks, it oozes, it gets places you don't want it
> to go. And serviceability is challenging. You need to pull the "wet" boards
> out, or you need to connect and disconnect fluid connectors, etc.  If
> you're in an environment where you can manage that (or are forced into it
> by necessity), then you can do it.
>
> --
>
> Dr Stuart Midgley
> sdm900 at gmail.com
>
>
>
> _______________________________________________
>
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
>
>
> _______________________________________________
>
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20181113/396ed47b/attachment-0001.html>