[Beowulf] Opteron cooling specifications?

Fri Apr 28 17:00:33 PDT 2006

I totally agree with Michael.

Except I'd assert that 32 nodes per rack is not a fully populated rack. :)  
Twenty-four of our 42U racks have 40 nodes apiece, plus 1U for a network 
switch and 1U blank.  It works fine for us and allows us comfortably fit 
1024 nodes in our pre-existing room.

At full power, one 40-node rack burns about 13kW.  Heat has not been a 
problem -- the only anomaly is that the topmost node in a 40-node rack 
tends to experience ambient temps a few degrees C higher than the others.  
That's presumably because some of the rear-vented hot air is recirculating 
within the rack back around to the front, and rising as it goes.  There may 
well be some other subtle airflow issues involved.

But yeah, initially Dell tried to tell me that 32 nodes is a full rack.  
Pfft. :)

In case it's of interest (and because I'm proud of our room :), our 
arrangement is:

A single 3-rack group for infrastructure (SAN, fileservers, master nodes, 
work nodes, tape backup, central Myrinet switch), placed in the middle of 
the room.

Four groups of 7 racks apiece, each holding 256 compute nodes and associated 
network equipment.  Racks 1-3 and 5-7 are 40-node racks (20 modes at the 
bottom, then 2U of GigE switch & blank space, then 20 nodes at the top).  
Rack 4 is 16 nodes at the bottom, a GigE switch in the middle, a Myrinet 
edge switch at the top, with quite a bit of blank space left over.

In the room, these are arranged in a long row:

[walkway][7racks][7racks][3racks][walkway][7racks][7racks][walkway]

And that *just* barely fits in the room. :)

One interesting element: Our switches are 48-port Nortel BayStack switches, 
so we have a natural arrangement: The 7-rack and 3-rack groups each have 
one switch stack.  The stacking cables go rack-to-rack horizontally between 
racks (only the end racks have side panels).

David

On Friday 28 April 2006 14:22, Michael Will wrote:
> A good set of specs according to our engineers could be:
>
> 1. No side vending of hot air from the case. The systems will be placed
> into 19" racks and there is no place for the air to go if it's blown
> into the side of the rack. Even if you take the sides off then you still
> will have racks placed next to each other. Airflow should be 100% front
> to back.
>
> 2. Along with that, there should be no "cheat holes" in the top, bottom
> or sides of the case. All "fresh" air should be drawn in from the front
> of the chassis. Again, the system will be racked in a 19" rack and there
> is no "fresh air" to be drawn in from the sides of the case (see 1
> above) nor will the holes be open when nodes are stacked one on top of
> the other in a fully populated rack (32 nodes per rack).
>
> 3. There should be a mechanical separation between the hot and cold
> sections of the chassis to prevent the internal fans from sucking in hot
> air from the rear of the chassis.
>
> 4. The power supply *must* vent directly to the outside of the case and
> not inside the chassis. The power supply produces approximately 20% of
> the heat in the system. That hot air must be vented directly out of the
> chassis to prevent it from heating other components in the system.
>
> 5. The system should employ fan speed control. Running high speed fans
> at less than rated speed prolongs their life and reduces power usage for
> the platform as a whole. Fan speed should be controlled by either
> ambient temperature or preferably by CPU temperature.
>
> 6. The system must have a way of measuring fan speed and reporting a fan
> failure so that failed fans can be replaced quickly.
>
> Michael Will / SE Technical Lead / Penguin Computing
>
> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org]
> On Behalf Of Bill Broadley
> Sent: Thursday, April 27, 2006 2:55 PM
> To: beowulf at beowulf.org
> Subject: [Beowulf] Opteron cooling specifications?
>
>
> I'm writing a spec for future opteron cluster purchases.  The issue of
> airflow came up.
>
> I've seen a surprising variety of configurations, some with a giant
> rotating cylinder (think paddle wheel), most with a variety of 40x28 or
> 40x56mm fans, or horizontal blowers.
>
> Anyone have a fan vendor they prefer?  Ideally well known for making
> fans that last 3+ years when in use 24/7.
>
> A target node CFM for a dual socket dual core opteron?
>
> A target maximum CPU temp?  I assume it's wise to stay well below the
> 70C or so thermal max on most of the dual core Opterons.
>
> Seems like there is a huge variation in the number of fans and total CFM
> from various chassis/node manufacturers.  A single core single socket 1u
> opteron I got from sun has two 40mm x 56mm, and 4 * 40mm x 28mm fans.
> Not bad for a node starting at $750.
>
> Additionally some chassis designs form a fairly decent wall across the
> node for the fans to insure a good front to back airflow.  Others seem
> to place fans willy nilly, I've even seen some that suck air sideways
> across the rear opteron.
>
> In any case, the nature of the campus purchasing process is that we can
> put in any specification, but can't buy from a single vendor, or award
> bids for better engineering.  So basically lowest bid wins that meets
> the spec.  Thus the need for a better spec.
>
> Any feedback appreciated.
>
> --
> Bill Broadley
> Computational Science and Engineering
> UC Davis
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org To change your subscription
> (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf