[Beowulf] Servers Too Hot? Intel Recommends a Luxurious Oil Bath

[Robert G. Brown]

On Wed, 5 Sep 2012, Gregory Matthews wrote:

> On 05/09/12 14:59, Ellis H. Wilson III wrote:
>> They save money because a) they can chill things much more easily (less
>> chillers), and moreover with a dense substance they can pipe it chilled
>> much farther (distance to chiller is far less important), unlike they
>> currently cannot with air.  So instead of a tall building with expensive
>
> er.. don't the chillers actually chill water/glycol rather than air?
> This is what is piped around...

Piped around through well-insulated pipes, in fact, much as it is at
institutions like Duke for the more mundane purpose of chilling air in
offices.  A good sized office building consumes many watts, and all of
it has to be removed to maintain working temperatures inside.

The point is that one doesn't have to be that close to the chillers --
it isn't a strong constraint on cluster growth.  Furthermore, I'd bet
that they can easily fully subscribe the capacity of the chillers they
have with simple geometries.  So when one talks about "adding pods" to
increase capacity, it is an unstated contingency that they also have to
"add chillers" as required in discrete increments as it is pretty easy
to build clusters bigger than one's chiller capacity no matter how the
cluster is arranged.  Clusters generate many times more power per unit
volume than the heart of a star, and you have surface to volume ratios
to consider for ANY geometry of cluster volume to chiller heat exchange
surface.  In the end, you have to remove 100% of the power generated by
the nodes plus the heat added to make the refrigerators run in accord
with the second law of thermodynamics against the ambient thermal
gradients.

Economically, what matters is being able to run more nodes per chiller
because one can minimize those thermal gradients and achieve a higher
mean coefficient of performance (higher heat capacity matters here) or
collect and transport the heat more efficiently (viscosity,
conductivity, turbulence matter here).  Oil (or Ethylene Glycol) has a
higher heat capacity than air but is much more viscous and expensive to
move.  Helium is cheap enough, EFFECTIVELY has a slightly lower heat
capacity than air but gains some benefit from lower viscosity and much
higher conductivity (it is actually less expensive to move -- that's why
you get that Donald Duck voice when speaking with a lungful of helium,
as I showed my kids using grocery store free balloons when they were
eight years old or thereabouts).  It is also dry and COTS heat sinks
will most definitely move helium where it is by no means clear that they
will move oil at all without overheating their coils.

Direct liquid cooling is completely different from direct gas cooling,
and COTS hardware is simply not designed for it.  It's not CLEAR that
even the use of helium is within engineering specs for COTS heat sinks
and fans, but there I'd be willing to bet that it is where for oil
immersion I'd bet that it is not.

    rgb

>
>
> -- 
> Greg Matthews        01235 778658
> Scientific Computing Group Leader
> Diamond Light Source Ltd. OXON UK
>
> -- 
> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu