[Beowulf] More about those underwater data centers

Sat Nov 17 00:35:44 PST 2018

Thats the kind of liquid cooling i was thinking of to be fair. yes it 
would be messy for maintenance but hell you probably can squeeze alot 
more performance out of the hardware no?

On 2018-11-13 16:31, John Hearns via Beowulf wrote:
> Ooooh…. liquid cooling video from sC18
> https://twitter.com/Yuryu/status/1062178413270786048
> 
> On Fri, 9 Nov 2018 at 03:42, Lux, Jim (337K) via Beowulf
> <beowulf at beowulf.org> wrote:
> 
>> I’ll bet the surface is rough enough that there are plenty of
>> nucleation centers.  Consider things like leads on parts.
>> 
>> FROM: Beowulf <beowulf-bounces at beowulf.org> on behalf of
>> "beowulf at beowulf.org" <beowulf at beowulf.org>
>> REPLY-TO: Prentice Bisbal <pbisbal at pppl.gov>
>> DATE: Thursday, November 8, 2018 at 7:47 AM
>> TO: "beowulf at beowulf.org" <beowulf at beowulf.org>
>> SUBJECT: Re: [Beowulf] More about those underwater data centers
>> 
>> One comment - my dissertation below is specifically about
>> non-ebullient immersion cooling. As Jim Lux pointed out in a later
>> e-mail, in ebullient cooling, some kind of surface feature to
>> promote nucleation could be beneficial. Ebbulient cooling is a whole
>> different beast from normal (non-ebullient) immersive cooling, since
>> in that case you have changes of state and gas bubbles flowing
>> through a liquid.
>> 
>> However, in all of the live and video demonstrations I've seen of
>> Novec, the processors were completely bare, bubbles were forming at
>> a pretty rapid rate, so again I think creating some sort of heat
>> sink for this would add cost with no significant benefit.
>> 
>> Prentice Bisbal
>> 
>> Lead Software Engineer
>> 
>> Princeton Plasma Physics Laboratory
>> 
>> http://www.pppl.gov
>> 
>> On 11/08/2018 10:40 AM, Prentice Bisbal wrote:
>> 
>> Heat fins are used to increase the surface area used for heat
>> transfer, since the rate of energy transfer by conduction is
>> directly proportional the surface area. Heat fins are needed when
>> air is involved because air has such a low thermal conductivity.
>> 
>> Thermal conductivity of liquids are much high, so heat fins aren't
>> as necessary. For example, I've read that water can transfer heat
>> orders of magnitude better than air, so using water to remove hear
>> from a processor would need orders of magnitude less surface area
>> for the same energy transfer rate.
>> 
>> Also, liquids have higher viscosities than gases, so we have to
>> worry about 'boundary layers'. A boundary layer is area where the
>> edge flowing fluid is in contact with a solid. The friction between
>> the liquid and the solid slows down the fluid near the solid. This
>> affects both gases and liquids, but since liquids have higher
>> viscosities, the effect is more noticeable.
>> 
>> Think about a car's radiator - the air side has all the fins on it,
>> and the liquid side has smooth pipe walls.
>> 
>> https://en.wikipedia.org/wiki/Boundary_layer
>> 
>> Convection heat transfer is an equally important mode of heat
>> transfer in fluids, and in the boundary layer, where the liquids
>> aren't moving as fast, heat transfer isn't as good, so you need to
>> keep your boundary layer from becoming too thick.
>> 
>> Since fluids have much higher thermal conductivities, and boundary
>> layer effects are more of a concern, I actually think a smooth heat
>> transfer surface would be better in these immersion cooling cases.
>> I'm sure smaller,  more spaced out fins would probably help heat
>> transfer without  creating too much of a boundary layer, but making
>> those heat sinks adds cost for increased performance in a situation
>> where it probably isn't needed.
>> 
>> Now direct-contact cooling systems like Asetek products  do have
>> fins on the liquid side, if I remember correctly, but that in those
>> systems, there are pumps to provide forced convection. In immersion
>> cooling, you are relying on natural convection, so there isn't as
>> much driving force to overcome viscosity/boundary layer effects to
>> force the liquid through the heat fins.
>> 
>> That's my thoughts, anyway.
>> 
>> Prentice
>> 
>> On 11/07/2018 04:12 AM, John Hearns via Beowulf wrote:
>> 
>> Thinking about liquid cooling , and the ebuillient cooling, the main
>> sources of heat on our current architecture servers are the CPU
>> package and the voltage regulators. Then the DIMMs.
>> 
>> Concentrating on the CPU die package, it is engineered with a flat
>> metal surface which is intended to have a thermal paste to transfer
>> heat across to a flat metal heatsink.
>> 
>> Those heatsinks are finned to have air blown across them to
>> transport the heat away.
>> 
>> In liquid immersion should we be looking at having a spiky surface
>> on the CPU die packages and the voltage regulators?
>> 
>> Maybe we should spray the entire board with a 'flocking'' compound
>> and give it a matt finish!
>> 
>> I am being semi-serious. I guess a lot of CFD simulation  done
>> regarding air cooling with fins.
>> 
>> How much work has gone into pointy surfaces on the die package,
>> which would increase contact area of course and also act as
>> nucleation points for bubbles?
>> 
>> One interesting experiment to do - assuming the flat areas of the
>> CPU in an immersive system do not have (non thermal paste) heatsinks
>> bolted on:
>> 
>> take two systems and roughen up the die package surfacewith
>> sandpaper on one. Compare temperatures.
>> 
>> ps. I can't resist adding this. Sorry Stu .
>> https://www.youtube.com/watch?v=kHnifVTSFEo
>> 
>> I guess Kenneth Williams is a typical vendor Site Engineer.
>> 
>> pps. the actress in the redress had her career ruined by this film -
>> she ver got a serious role again after perfectly being typecast.
>> 
>> On Tue, 6 Nov 2018 at 22:57, Prentice Bisbal via Beowulf
>> <beowulf at beowulf.org> wrote:
>> 
>> On 11/06/2018 02:03 PM, Lux, Jim (337K) wrote:
>> 
>> True enough.
>> 
>> Ebullient cooling does have some challenges – you can form vapor
>> films, which are good insulators, but if you get the system working
>> right, nothing beats phase changes for a heat transfer.
>> 
>> If I recall what I learned in my Transport Phenomena classes in
>> engineering school, you need a reasonably high temperature
>> difference to get a stable film like that. For that to happen,
>> radiant heat transfer needs to be the dominant heat transfer
>> mechanism, in the range of operation we are talking about, the
>> temperature difference isn't that great, and conduction is still the
>> dominant form of heat transfer.
>> 
>> Here's an example of what 3M Novec ebullient cooling looks like. It
>> doesn't look like it's anywhere near the film boiling regime:
>> 
>> https://www.youtube.com/watch?v=CIbnl3Pj15w
>> 
>> --
>> Prentice
>> 
>> FROM: Beowulf [mailto:beowulf-bounces at beowulf.org] ON BEHALF OF
>> Prentice Bisbal via Beowulf
>> SENT: Tuesday, November 06, 2018 8:17 AM
>> TO: beowulf at beowulf.org
>> SUBJECT: Re: [Beowulf] More about those underwater data centers
>> 
>> . And serviceability is challenging. You need to pull the "wet"
>> boards out, or you need to connect and disconnect fluid connectors,
>> etc.  If you're in an environment where you can manage that (or are
>> forced into it by necessity), then you can do it.
>> 
>> I think everyone on this list already knows I'm no fan of mineral
>> oil immersion (It just seems to messy to me. Sorry, Stu), but
>> immersion cooling with other liquids, such as 3M Novec engineered
>> fluid addresses a lot of your concerns. It as a low boiling point,
>> not much above room temperature, and it was originally meant to be
>> an electronic parts cleaner (according to a 3M rep at the 3M booth
>> at SC a few years ago, so if you pull a component out of it, it
>> dries very quickly and should be immaculately clean.
>> 
>> The low boiling point is an excellent feature for heat transfer,
>> too, since it boils from the heat of the processor (ebullient
>> cooling). This change of state absorbs a lot of energy, making it
>> very effective at transferring heat away from the processor. The
>> vapor can then rise and condense on a heat exchanger with a chilled
>> water heat exchanger, where it again transfers a lot of heat through
>> a change of state.
>> 
>> Prentice
>> 
>> On 11/05/2018 06:30 PM, Stu Midgley wrote:
>> 
>> I refute both these claims.
>> 
>> You DO want to run your boards immersed in coolant.  It works
>> wonderfully well, is easy to live with, servicing is easy... and
>> saves you almost 1/2 your power bill.
>> 
>> People are scared of immersion cooling, but it isn't that difficult
>> to live with.  Some things are harder but other things are way
>> easier.  In total, it balances out.
>> 
>> Also, given the greater reliability of components you get, you do
>> less servicing.
>> 
>> If you haven't lived with it, you really have no idea what you are
>> missing.
>> 
>> Serviceability is NOT challenging.
>> 
>> You really do NOT want to run boards immersed in coolant - yeah,
>> there's folks doing it at HPC scale
>> 
>> Whatever the coolant, it leaks, it oozes, it gets places you don't
>> want it to go. And serviceability is challenging. You need to pull
>> the "wet" boards out, or you need to connect and disconnect fluid
>> connectors, etc.  If you're in an environment where you can manage
>> that (or are forced into it by necessity), then you can do it.
>> 
>> --
>> 
>> Dr Stuart Midgley
>> sdm900 at gmail.com
>> 
>> _______________________________________________
>> 
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
>> Computing
>> 
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf [1]
> 
> _______________________________________________
> 
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> Computing
> 
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> 
>  _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> 
> 
> Links:
> ------
> [1] http://www.beowulf.org/mailman/listinfo/beowulf
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin 
> Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf