[Beowulf] More about those underwater data centers
Prentice Bisbal
pbisbal at pppl.gov
Thu Nov 8 07:46:08 PST 2018
One comment - my dissertation below is specifically about non-ebullient
immersion cooling. As Jim Lux pointed out in a later e-mail, in
ebullient cooling, some kind of surface feature to promote nucleation
could be beneficial. Ebbulient cooling is a whole different beast from
normal (non-ebullient) immersive cooling, since in that case you have
changes of state and gas bubbles flowing through a liquid.
However, in all of the live and video demonstrations I've seen of Novec,
the processors were completely bare, bubbles were forming at a pretty
rapid rate, so again I think creating some sort of heat sink for this
would add cost with no significant benefit.
Prentice Bisbal
Lead Software Engineer
Princeton Plasma Physics Laboratory
http://www.pppl.gov
On 11/08/2018 10:40 AM, Prentice Bisbal wrote:
>
> Heat fins are used to increase the surface area used for heat
> transfer, since the rate of energy transfer by conduction is directly
> proportional the surface area. Heat fins are needed when air is
> involved because air has such a low thermal conductivity.
>
> Thermal conductivity of liquids are much high, so heat fins aren't as
> necessary. For example, I've read that water can transfer heat orders
> of magnitude better than air, so using water to remove hear from a
> processor would need orders of magnitude less surface area for the
> same energy transfer rate.
>
> Also, liquids have higher viscosities than gases, so we have to worry
> about 'boundary layers'. A boundary layer is area where the edge
> flowing fluid is in contact with a solid. The friction between the
> liquid and the solid slows down the fluid near the solid. This affects
> both gases and liquids, but since liquids have higher viscosities, the
> effect is more noticeable.
>
> Think about a car's radiator - the air side has all the fins on it,
> and the liquid side has smooth pipe walls.
>
> https://en.wikipedia.org/wiki/Boundary_layer
>
> Convection heat transfer is an equally important mode of heat transfer
> in fluids, and in the boundary layer, where the liquids aren't moving
> as fast, heat transfer isn't as good, so you need to keep your
> boundary layer from becoming too thick.
>
> Since fluids have much higher thermal conductivities, and boundary
> layer effects are more of a concern, I actually think a smooth heat
> transfer surface would be better in these immersion cooling cases. I'm
> sure smaller, more spaced out fins would probably help heat transfer
> without creating too much of a boundary layer, but making those heat
> sinks adds cost for increased performance in a situation where it
> probably isn't needed.
>
> Now direct-contact cooling systems like Asetek products do have fins
> on the liquid side, if I remember correctly, but that in those
> systems, there are pumps to provide forced convection. In immersion
> cooling, you are relying on natural convection, so there isn't as much
> driving force to overcome viscosity/boundary layer effects to force
> the liquid through the heat fins.
>
> That's my thoughts, anyway.
>
> Prentice
> On 11/07/2018 04:12 AM, John Hearns via Beowulf wrote:
>> Thinking about liquid cooling , and the ebuillient cooling, the main
>> sources of heat on our current architecture servers are the CPU
>> package and the voltage regulators. Then the DIMMs.
>> Concentrating on the CPU die package, it is engineered with a flat
>> metal surface which is intended to have a thermal paste to transfer
>> heat across to a flat metal heatsink.
>> Those heatsinks are finned to have air blown across them to transport
>> the heat away.
>>
>> In liquid immersion should we be looking at having a spiky surface on
>> the CPU die packages and the voltage regulators?
>> Maybe we should spray the entire board with a 'flocking'' compound
>> and give it a matt finish!
>> I am being semi-serious. I guess a lot of CFD simulation done
>> regarding air cooling with fins.
>> How much work has gone into pointy surfaces on the die package, which
>> would increase contact area of course and also act as nucleation
>> points for bubbles?
>>
>> One interesting experiment to do - assuming the flat areas of the CPU
>> in an immersive system do not have (non thermal paste) heatsinks
>> bolted on:
>> take two systems and roughen up the die package surfacewith sandpaper
>> on one. Compare temperatures.
>>
>> ps. I can't resist adding this. Sorry Stu .
>> https://www.youtube.com/watch?v=kHnifVTSFEo
>> I guess Kenneth Williams is a typical vendor Site Engineer.
>> pps. the actress in the redress had her career ruined by this film -
>> she ver got a serious role again after perfectly being typecast.
>>
>>
>>
>>
>>
>>
>>
>> On Tue, 6 Nov 2018 at 22:57, Prentice Bisbal via Beowulf
>> <beowulf at beowulf.org <mailto:beowulf at beowulf.org>> wrote:
>>
>> On 11/06/2018 02:03 PM, Lux, Jim (337K) wrote:
>>>
>>> True enough.
>>>
>>> Ebullient cooling does have some challenges – you can form vapor
>>> films, which are good insulators, but if you get the system
>>> working right, nothing beats phase changes for a heat transfer.
>>>
>> If I recall what I learned in my Transport Phenomena classes in
>> engineering school, you need a reasonably high temperature
>> difference to get a stable film like that. For that to happen,
>> radiant heat transfer needs to be the dominant heat transfer
>> mechanism, in the range of operation we are talking about, the
>> temperature difference isn't that great, and conduction is still
>> the dominant form of heat transfer.
>>
>> Here's an example of what 3M Novec ebullient cooling looks like.
>> It doesn't look like it's anywhere near the film boiling regime:
>>
>> https://www.youtube.com/watch?v=CIbnl3Pj15w
>>
>> --
>> Prentice
>>
>>> *From:*Beowulf [mailto:beowulf-bounces at beowulf.org] *On Behalf
>>> Of *Prentice Bisbal via Beowulf
>>> *Sent:* Tuesday, November 06, 2018 8:17 AM
>>> *To:* beowulf at beowulf.org <mailto:beowulf at beowulf.org>
>>> *Subject:* Re: [Beowulf] More about those underwater data centers
>>>
>>> . And serviceability is challenging. You need to pull the
>>> "wet" boards out, or you need to connect and disconnect
>>> fluid connectors, etc. If you're in an environment where
>>> you can manage that (or are forced into it by necessity),
>>> then you can do it.
>>>
>>> I think everyone on this list already knows I'm no fan of
>>> mineral oil immersion (It just seems to messy to me. Sorry,
>>> Stu), but immersion cooling with other liquids, such as 3M Novec
>>> engineered fluid addresses a lot of your concerns. It as a low
>>> boiling point, not much above room temperature, and it was
>>> originally meant to be an electronic parts cleaner (according to
>>> a 3M rep at the 3M booth at SC a few years ago, so if you pull a
>>> component out of it, it dries very quickly and should be
>>> immaculately clean.
>>>
>>> The low boiling point is an excellent feature for heat transfer,
>>> too, since it boils from the heat of the processor (ebullient
>>> cooling). This change of state absorbs a lot of energy, making
>>> it very effective at transferring heat away from the processor.
>>> The vapor can then rise and condense on a heat exchanger with a
>>> chilled water heat exchanger, where it again transfers a lot of
>>> heat through a change of state.
>>>
>>> Prentice
>>>
>>> On 11/05/2018 06:30 PM, Stu Midgley wrote:
>>>
>>> I refute both these claims.
>>>
>>> You DO want to run your boards immersed in coolant. It
>>> works wonderfully well, is easy to live with, servicing is
>>> easy... and saves you almost 1/2 your power bill.
>>>
>>> People are scared of immersion cooling, but it isn't that
>>> difficult to live with. Some things are harder but other
>>> things are way easier. In total, it balances out.
>>>
>>> Also, given the greater reliability of components you get,
>>> you do less servicing.
>>>
>>> If you haven't lived with it, you really have no idea what
>>> you are missing.
>>>
>>> Serviceability is NOT challenging.
>>>
>>> You really do NOT want to run boards immersed in coolant
>>> - yeah, there's folks doing it at HPC scale
>>>
>>> Whatever the coolant, it leaks, it oozes, it gets places
>>> you don't want it to go. And serviceability is
>>> challenging. You need to pull the "wet" boards out, or
>>> you need to connect and disconnect fluid connectors,
>>> etc. If you're in an environment where you can manage
>>> that (or are forced into it by necessity), then you can
>>> do it.
>>>
>>> --
>>>
>>> Dr Stuart Midgley
>>> sdm900 at gmail.com <mailto:sdm900 at gmail.com>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> Beowulf mailing list,Beowulf at beowulf.org <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>>>
>>> To change your subscription (digest mode or unsubscribe) visithttp://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>>
>>
>> _______________________________________________
>> Beowulf mailing list,Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visithttp://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20181108/8480d490/attachment-0001.html>
More information about the Beowulf
mailing list