[Beowulf] followup on 1000-node Caltech cluster
sdm900 at gmail.com
Sat Jun 18 20:15:15 PDT 2005
An interesting system.
We have had issues with air-cooling racks that generate more than
about 8kW of energy. We just couldn't get enough cold air into the
front of the rack. We also had issues of air from the back of the
rack recirculating around to the front of the racks (on the edges of
the cluster)... so we had to build baffles to try and prevent this
circulation. I will be intersted to see how you go.
On 19/06/2005, at 7:04, David Kewley wrote:
> Our compute node racks will dissipate up to ~14kW. The power
> strips are
> three 5.7kW 208V 3-phase APC networked/switched "rack PDUs". The
> intakes for the HVAC units are ~3 feet from the backs of the
> racks. We
> expect to have almost laminar air flow, but with good local & roomwide
> mixing due to an up&down airflow pattern from the supply ducts.
When we lost air conditioning to our machine room (we currently have
about 300kW of gear) the room went up about 10 degrees celcius in
about 10 minutes. Now, we have automatic processes to shut machines
down once they get to 40 degrees for a minute. BUT the problem isn't
the absolute temperature, its the rate of rise that kills gear. Even
though our machine hadn't shut down (it didn't reach the critical
temperature) we were loosing disks and dimms all over the place just
due to the sudden rate of rise of temperature. When we spoke with HP
and check out contract with them, it was quite clearly stated that
the environment wasn't to go up more then 10 degrees in less than an
Your machine sounds a lot larger than our old one... so I expect your
problems to be worse.
> It was calculated that if the air cooling suddenly stopped while the
> power into the room was 400kW, but the fans kept the air circulating,
> the average rate of temperature increase would be 4 degrees F per
> second. I remain a bit skeptical of that number personally -- for
> example, there was no allowance for the heat capacity of the
> now-stationary chilled water in the heat-transfer coils, nor of the
> walls, floors, ceiling, air ducts, and other surfaces in the room.
> the Planetary Science folks here know their atmostpheric modeling
> what they do for a living), so I'm pretty confident that their results
> are correct given their simple assumptions.
Good luck with it all
Dr Stuart Midgley
sdm900 at gmail.com
More information about the Beowulf