[Beowulf] Servers Too Hot? Intel Recommends a Luxurious Oil Bath

[Robert G. Brown]

On Wed, 5 Sep 2012, Lux, Jim (337C) wrote:

>
>
> On 9/4/12 11:54 AM, "Ellis H. Wilson III" <ellis at cse.psu.edu> wrote:
>
>> On 09/04/2012 12:54 PM, Lux, Jim (337C) wrote:
>>> Hydrogen is cheaper than He and works even better.  Just make sure you
>>> don't have any air leaks in (i.e. keep a bit of positive pressure).  For
>>> the "server farm in a container" model, this would work just fine..
>>> leaks would just float up into the atmosphere.
>>
>>
>> I know we've been taking things to the uber-scale level with this
>> conversation, but does anyone have suggestions for small (homebrew
>> Beowulf) clusters?  I've considered oil before, but for all the
>> capillary concerns voiced in this list have avoided it.  I would
>> consider a reasonable gas (NOT hydrogen) if one could be suggested along
>> with a feasible way to keep that gas in a small rack or similar
>> structure, or an alternative to oil if a nicer one (albeit not as
>> efficient) could similarly be suggested.  Perhaps air or piped
>> water-cooling is indeed my best bet.
>>
>
> I've thought that some form of spray cooling might work, with a dramatic
> reduction in the amount of oil/fluorinert/silicone oil, etc.

Yes, but then if it is oil it is aerosolized.  Not just a fire, an
actual explosion!  Even relatively inert materials burn like hell if you
get surface to volume large enough (for small enough droplets).

And you still have to deal with wet, icky hardware for service.  And
with toxicity issues and environmental regulations galore.

Even helium isn't a clear slam-dunk; its heat capacity and conductivity
are much higher than air, but its MOLAR heat capacity is actually less.
Air is roughly c_p = 1 kJ/kg, helium is c_p = 5 kJ/kg, but the molar
mass of air is 29, the molar mass of helium is 4, so at STP you actually
have more heat capacity in a pod full of air than you do in a pod full
of helium.

So then you are down to comparing conduction to convection and picky
details of heat transfer at the heat sink surfaces.  Helium has much
better conductivity, but you have to trade it off against somewhat lower
heat capacity per cubic meter at STP, different Reynolds number,
different viscosity...

Helium for air MIGHT still be a cost-beneficial swap; higher
conductivity increases the thickness of the surface layer receiving heat
at heat exchange surfaces, very likely in a way proportional to the
conductivity.  Thus helium in spite of having a slightly lower
volumetric heat capacity compared to air might absorb heat in several
times more volume in the vicinity of the heat sinks in a given amount of
time, and with lower molecular mass helium would be moved across those
sinks much faster as well (although possibly with less turbulence!).
This would result in much more uniform heating of the helium and hence
overall lower temperatures with less of a thermal gradient required to
remove all of the heat from the chips and spread it around in the
helium.

Quite seriously, I would back of the envelope estimate a factor of
perhaps 4 or 5 increased cooling efficiency of the heat sinks, although
to get the highest benefit one would have to redesign the heat sinks
themselves to take advantage of the physical properties of the helium
and ensure that you still have e.g. turbulent versus laminar flow (or
vice versa) where and as desired.  This doesn't really save you any
money, mind you -- you still have to remove 100% of the power delivered
to the cluster 100% of the time to maintain steady operating
temperatures -- but it might lower that operating temperature at
constant cost (or let you lower the operating temperature at lower cost
than alternatives to achieve higher clock rates).

Would the savings compensate for the enormous bump in maintenance costs
compared to just walking into a pod and swapping an open-top system in
air?  Human skilled labor costs at least $50/hour, more likely twice
that.  Human labor either working with oxygen masks in a sealed helium
environment or partially or fully shutting down pods in order to pump
out the helium and replace it with air would definitely double or even
triple that.  But even at $50/hour, you're talking five or six minutes
to swap a system now -- walk in, flip a switch, unplug/untether the open
top box, plug/tether the replacement, flip a switch, walk out --
compared to what, put on a suit, cycle through an airlock, then do the
same thing or power the cluster partially down, scavenge the helium,
open the pod to air, go in and do your work, reseal it, pump out the
air, replace the helium, power up (fully) again?  Or pay somebody to go
into the pod and just wear an oxygen mask all day to be on hand to do
replacements?  It looks like you're comparing human cost of $5 to $10
for a node replacement to something at least 3-4 times that, on nodes
that themselves cost only $200 to $300, for a benefit measured at
ballpark $1/watt/year in net power reduction.  Very dicey, no clear
winner.

MAYBE one could work out a way where it's CBA-positive, but I'd want to
see the entire cost-benefit analysis and real data drawn from a
prototype helium-filled sealed pod with an airlock to be able to get a
real-world estimate of the additional costs in pod construction,
additional benefits in cooling cost and/or clock or some other
"valuable" operational metric, additional costs in labor and operation
before I came close to believing it.  It barely justifies building such
a prototype -- maybe -- if more precise computations than my estimates
suggest that there could be a real benefit.

    rgb

>
>
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu