[Beowulf] 'liquid cooled' racks

[Bruce Allen]

Hi Mark,

>> For my next cluster room, I am hoping to use 'liquid cooled' racks make by 
>> Knurr (CoolTherm, http://www.thermalmanagement.de/).  The scale is 67 racks 
>> x 7.5kW heat removal per rack (36 U usable per rack).
>
> 7.5 KW/rack isn't much; are you designing low-power nodes?

Yup -- I guess so.  For example our current cluster nodes are dual core 
Opteron 175s (Supermicro H8SSL-i motherboards).  They cost about $1200 for 
2 x 2.2 GHz, with 1GB per core, and use 180 W under load (per 1U).

I will have enough rack space (67 x 34 U, per 500kW power and cooling) 
that if I want to use nodes that dissipate 300 W per node then I will 
simply limit myself to 25 nodes per rack.

>> The racks have front and back doors that close, and contains fans in the 
>> back which circulate air through a heat exchanger located in the bottom of 
>> the rack.  The heat exchanger transfers the heat into chilled water.
>
> APC has something similar with the cooling on the side.

Can you give a positive or negative opinion about either the Knurr or APC 
racks?  Have you used them yourself in a system?

>> The advantages of this are that it is quiet, and that you don't need 
>> vertical height for underfloor ducting or overhead hot air removal. 
>> Disadvantages are cost, potential difficulty of working within the rack. 
>> and loss of one rack of cooling capacity if you open both the front and 
>> back doors.
>
> I'm pretty skeptical of the sealed-pod approach, since it seems to 
> multiply the number of parts, create access issues, doesn't seem to 
> actually save on space, etc.

It does save on vertical space, and it reduces noise to the level where 
you can work comfortably in the room.

> I've also been burned by cold water cooling, so to speak (assuming you 
> don't have your own, well-controlled CW plant.)

I'll have my own self-contained plant, designed by German engineers.  They 
seem to know their stuff.

> I would definitely consider a normal big-chillers approach _with_
> back-of-rack CW boosters (heat exchangers).

Can you provide a URL or recommendation for these back-of-rack CW 
boosters?

> and I'd definitely consider creative layouts of racks and chillers (for 
> instance, I'm not crazy about the hot/cold-aisle approach - something 
> W-shaped would be better for my ~50-rack machineroom.  or even just 
> lining all the hot racks up against the chillers along one wall.)
>
>> (In this case I have access to 'building' funds that can not be used to buy 
>> more cpus, so the cost issue is not important.)
>
> are you sure you can't just be clever about laying out normal front-to-back 
> racks?  possibly with back-of-rack heat-exchangers?

I'd like a URL for the back-of-rack heat exchangers, please!

(PS: though that approach still sounds noisy!)

Cheers,
 	Bruce