[Beowulf] let's standardize liquid cooling

[Lux, Jim (337C)]

I have a modest proposal: 

>> That's always a tricky phrase.. fortunately you're not talking about novel food sources here.

standardize the location of liquid-cooled cold plates in each 19" rack. 
nodes would have internal heatpipes from heat sources (presumably CPUs
mostly) to plates along the sides to mate/contact with the rack.

I have an aging machineroom with ~50 racks, with the compute racks dissipating something like 11KW aircooled.  total dissipation a little under 300KW.  contemplating upgrades, it would need to go to about 1MW to be viable, and should be smart about cooling (free cooling, or pre-heating the building's air intake during winter.)  our current setup is reasonably tuned (well-partitioned air, PUE of 1.3-1.4.)

we happen to be located in a southerly part of Canada, and would definitely need active cooling during summer.  but the real point of using liquid cooling right to the CPU is that it would reduce the overall thermal resistance, and permit higher outgoing temperatures, which then stand more of a chance of being interesting, utility-wise.

it seems to me like this would be doable - vendors currently each have a slightly different node-rail combination, but I don't think this is really perceived as a competitive advantage.  they would presumably have to accept a standard design (perhaps just a literal rail with no ball-bearing widgets to vertically position the node.  I personally find little or no value in the ability to pull out a node and have it hang in place.

> for test equipment racks with heavy RF equipment, we use pieces of 1.5x1.5" or2x 2" angle aluminum (or steel) as a sort of shelf along each side. The problem is always whether there is enough space between the chassis, though.  If your 4 U box is exactly 7" tall, then you can't stack them tightly. If, on the other hand, there were feet on the bottom, and the "rack ears" are the only thing that is 7", it can work.  But there's still the problem that there's no standard "rubber feet clearance" (or, more properly, the distance between the bottom edge of the rack ear/front panel fascia and the bottom of the chassis.  Shimming is a pain, and even more of a pain is "drill the holes a bit over size and slide the angle bracket up and down until it's at the right height".   

>If you're going to promulgate a new packaging approach,  I think we should promote a standard that uses something like bakery carts with sheet pans.  Why use 10-32 screws into weird spaced holes, just because it was convenient for the phone company 75 years ago?.  Promote a standard that has horizontal steps on which a matching sheet metal step sits. The step should be on the order of 1-3 cm high, so that you can make something stamped out of sheet metal that is strong with ribs (the "shelf" doesn't have to be continuous). To keep the thing from sliding out, you could use either a couple bolts at either end, a latch, or a bar that runs down the side and holds all slides in.

I don't know whether there would have to be some kind of clamping mechanism to put pressure on the node, improving the plate-to-plate contact.
I guess that would need to be per-U, which is a bit of a pain.

>  Look at "wedge-locks" as used in conduction cooled electronic equipment.  There's a bar with a diagonal cut in it, and a bolt serving as a drawbar that forces the two pieces apart. But, is getting the heat out through the side panel of the box what you really want? A proper conduction cooled design is difficult. 

> for something a bit more novel, and with very good thermal properties, there's a carbon fiber flocking.   Imagine the surface covered with short carbon fibers all standing on end (epoxy adhesive holds them to the plate).  It mates with a similar plate, and because there are millions of fiber to fiber contacts, the thermal resistance is quite low.  (Carbon fibers have very good thermal conductivity).  It tolerates misalignment and non-flat surfaces (a real problem with any heat transfer by conduction).  You don't need high clamping forces, because the contact area between fibers is large.

>BTW, clamping and  ceramic packages is a dicey proposition: the package is brittle. A tiny misalignment, and you crack the package.



internally, a node could use heatpipes or possibly small pumps.
how the rack-mounted plates are cooled would be available for innovation (in-rack DX cooling might be attractive, though for bigger installations, presumably some center-wide circulation of glycol/etc would make sense.)

there is some vendor-specific activity along these lines, and a fairly long history of per-rack heat exchangers.  obviously, avoiding vendor lockin is hugely attractive, especially to a Beowulf mindset.  vendors, of course love lockin, but have accepted standards in various ways (Intel's PSU standards, JEDEC ram, etc)

> funny you should mention JEDEC RAM, because there WAS a huge IP battle, with scurrilous activities on the part of vendors participating in the consortium holding submarine patents, etc..