[Beowulf] let's standardize liquid cooling

[Mark Hahn]

> Sounds expensive, complicated, and challenging.

I donno - it seems elegantly modular to me.  vendors are responsible
for getting the heat to the cold plate (via heatpipes, probably.  these
days, heatpipes are extremely widespread and well-controlled.  every
laptop has them, many GPU cards and desktop heatsink/fan units.)
and the facility is responsible for extracting heat from the cold plates.

successful standards are sort of like contracts, where participants 
get an easy-to-understand responsibility.  cooling by air works pretty
well, since the facility provides a cold aisle.

> How about a MUCH simpler proposal: eliminate fans from compute nodes.

simpler, yes, and it's been done by some vendors.  the problem is that 
air sucks for moving heat.  I'd like to reject heat from the datacenter
at useful temperatures.  to do that requires low thermal resistance,
and that's simply not possible with air (unless we switch to some new 
semiconductor that runs happily at higher temperature.)

> Nodes should:
> * assume good front to back airflow

that's fine, as long as your cold aisle is 15C or so.  your hot aisle
will be 30C or so.  not useful for anything.  I gather that some sites
have managed to tweak to 20/40, but that's not much different.

> Racks would:
> * have large fans front AND back that run at relatively low
>  rpm, and relatively quiet.

well, one of the good things about conventional design is that nodes 
simply help themselves to as much cold air as they need.  doesn't work
perfectly, but individual feedback loops are good for clusters where 
nodes vary a lot.  (OTOH, HPC clusters should be 100% all the time...)

> My measurements of 1U nodes have shown a significant fraction of the
> power budget goes to fans!

oh, sure, not really news.  I was surprised as well, when I installed 
a cluster of nodes which dissipated 250W, except when the 14 onboard 
3.5W muffin fans ramp up to 14krpm, helpfully providing an extra 50W
just when you need it least ;)

> Fans that fit in 1U (even 2U) nodes seem to
> generate more noise, heat, and vibration than airflow.

the ones in DL145g2 generate lots of airflow.  that's not to say
I wouldn't prefer rack-based blowers and per-node active luvers.
managing airflow in a basic DC is also not

the problem is I want 2-4x the heat density with better PUE,
preferably lower than 1.0 by using the heat for something...

regards, mark hahn.