Fw: Re: [Beowulf] cooling question: cfm per rack?

Fri Feb 11 14:59:55 PST 2005

Mike,

I've been trying to pick the brains of other folks on the
beowulf list who have computer rooms with modern equipment.

One problem with the existing air, with regards to future
expansion, is apparently the total amount of air that the
current A/C can move.  This is all horrendously complicated
and needs to be looked at carefully by a HVAC consultant.
Pretty sure we have enough tons and flow for now, meaning
my rack and Deshaies and everything else I know is going in there
in a couple of months.  More and more convinced that we don't
have enough to handle multiple full racks of the next generation
of computers.

Jim Lux from JPL answered my questions as attached after
my signature.  His back of the envelope
calculations for a 5kW rack (roughly 
equal to what I have now) give a requirement for 1800 cfm flow through
the rack.  The current A/C is, according to the A/C guy who was
here, good for only 5500 cfm.  However, since I don't know what the
inlet or outlet temperatures on the rack are going to be (ie,
the temperature of the air the A/C returns to the room and how
hot the air is coming out the back) the required cfm may be
quite different than this.  Hmm, let me go borrow a thermometer
and measure it, 22 C in, 32 C out the back, on the node
in the middle of the  rack.  So there's 10 degrees across my rack
and he assumes 15.  Anyway, a safer estimate for the next generation
is 10kW, and there are people who predict 20kW, so total
airflow through the A/C seems unlikely to be sufficient a few
years down the road.  Assuming that people put this equipment 
in the room.  

Sorry, to be vague, there are just so many unknowns.

I also talked to Darryl Willick, who runs a bunch of machine rooms
on campus for Chemistry and some of Rees, Bjorkman and Mayo's
stuff.  His main room is about at capacity now with
6 full racks and a few odds and ends.  He has 2 x 250A panels
in there and apparently only a 45kW A/C unit.  That second
number is really odd because they aren't usually rated that
way, but that's the number he remembered. If he's right that's
45000/3500=12 tons, roughly the same as the unit currently
in the Rees area.  He said his had to be serviced
recently because they were having overheating problems, but only
a belt was changed.  Unknown how many cfm it is.  He has a small
workstation area that is somehow or other connected to his machine
room ventilation wise, and apparently when they prop the door open
in the workstation area it causes problems in the machine room.
So maybe it would make sense to put a small separate A/C unit
in the proposed classroom to avoid those sorts of complications
in the future.  Or maybe it can tap off building air.

Darryl did say something interesting though, he said that for
some units the A/C people can increase the capacity by changing
the pulleys around.  Apparently this blows more air, and the
cold water isn't limiting, so it effectively upgrades the unit
without changing very much.  Darryl said that this was done
at some point for Mayo's computer room in the subbasement
of the BI.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech

------------- Forwarded message follows -------------

At 08:17 AM 2/11/2005, you wrote:
>In designing a computer room two key factors are:
>
>1.  Power in   (electricity)
>2.  Power out  (A/C)
>
>The second term really has two parts:
>
>   A.  the amount of air moved
>   B.  the reduction in temperature of that air across the A/C unit
>
>The latter part is specified in tons.  The A/C guys I've spoken
>with recently utilize some more or less standard relationship
>between cubic feet per minute (cfm) and A/C tons for the units they
>maintain.  These run off the campus cold water supply, so
>it makes sense that heat out is proportional to flow across, assuming
>that the cold water has a very large heat capacity.
>
>However, in terms of cooling the units themselves, the amount of
>air flow through the racks is also important.  That flow is
>also in cfm.  Ideally cfm through the racks would be equal to cfm
>through the A/C, ie, all air goes once through the racks and then
>directly through the A/C.  Even more ideally cfm through _each_ rack
>could be modulated somehow, since some racks move much more
>air than others and putting a low flow rack next to a high flow rack
>might drive the air the wrong way through the low flow unit.
>
>How does one calculate an optimal cfm through a rack?

Decide on a maximum outlet temperature (say, 30C)
Find your inlet air temperature (say, 15C)
You know your dissipation.. (say, 5kW)

Calculate how much air you need to move using the specific heat of air.
(about 1 kJ/(kg K))

5 kJ/sec means you'd need 5 kg/sec for a 1 degree rise, but here, with a 15 
degree rise, you can get by with .33 kg/sec.  Turn the kg/sec into cfm... 
.33 kg * 1.3 m3/kg = .43
cubic meters/sec.  There's about 35 cubic feet in a cubic meter, so we need 
about 15 cubic feet per second.  Multiply by 60 and you get a bit more than 
900 cfm.

Now.. that's idealized, so double it.  1800 cfm or so.

Step 2: How big is the duct?  Generally, you don't want to go any faster 
than 1000 linear feet per minute, so your duct will need to be about 2 
square feet.  (you begin to see why you don't want some little 6" diameter 
blower...)

>For a specific example with round numbers, let's say it's a
>25U rack, dissipates 10kW, and has a single 50 cfm per minute output
>fan per 1U node.  (Ie, all air out must go through that path.)
>
>There seem to be a bunch of variables that are hard to deal with.
>For instance, adding the exhaust fans would be 50*25 = 1250 cfm.
>Is that all there is to it? But that type of fan only runs at
>the stated flow rate if the pressures are exactly as specified.
>Without incredibly careful balancing of the pressure across the
>rack it won't generally run at 50 cfm.

This is precisely the case.  And, of course, the actual circumstances will 
be nothing like what the design specs are.

>Is cfm the key unit here or should one think in terms of pressure
>at various points in the room?

Trying to come up with an accurate aerodynamic model is a worthy challenge 
for a very large cluster (computational challenge, not thermal).

It's all done by rules of thumb and adding lots of margin.

Use the rough sizing technique to get an approximate air flow.  Use 
reasonable sized ducts and air speeds. Measure the actual outlet
temperatures.

Actually, what most people do is a rough sizing, then call in someone who 
actually does this for a living (a HVAC contractor) and use their rough 
sizing to validate what the contractor tells you you should have.

>Thanks,
>
>David Mathog
>mathog at caltech.edu
>Manager, Sequence Analysis Facility, Biology Division, Caltech
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit 
>http://www.beowulf.org/mailman/listinfo/beowulf

James Lux, P.E.
Spacecraft Radio Frequency Subsystems Group
Flight Communications Systems Section
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875