high physical density cluster design -structural...

Tue Mar 6 12:33:02 PST 2001

On Tue, Mar 06, 2001 at 03:12:42PM -0500, David Grant's all...
> This has been an interesting thread, but I do have a concern about
> appropriate cooling  with "homegrown" 1U chassis.  Yes, you can build a box
> the will physically support the hardware in a 1U form factor.  My concern
> would be long term, and not so long term heat related failures on CPU's
> and/or disk drives.....
> 
> just my .02

agreed, totally. We have no hardrives, which are far more stressed by hot
environments considering the moving parts and what not. The cpu I am not so
worried about, since electron migration path burning takes a long time to
occur at high temperatures. Nonetheless, we are going to keep things very
cool. I intend to remain several (10? 15?) degrees C lower than the max
operational temp (where we start seeing machines crash). With Durons I've seen
this around 45C. We have full access to a VERY large airflow direct from the
Liebert (as I said, an 8" pipe with a 20 or 30 mph (my guess) cold airflow
coming out of it - which comes to something like 2500 to 3500 CFM of air at
67F) so I think with careful construction we'll be able to dissipate this
amount of heat.

The liebert system was designed to provide 20 tons of cooling over 4000 sq
feet, but the density of machines installed is lower than we planned for
(customers put in fewer boxes than we expected, its up to them). We can
therefore divert more airflow to this room if we really need to (a few of
the rooms are still completely empty as well).

If anyone thinks that even with these considerations this is foolish,
let me know. :)

I mean, if we have problems we can always just seperate the boards by
larger amounts and move the stacks apart. I am sure 48 or 64 boards
in the one room will be fine - other customers have this density
are fine. Its not even a full degree C warmer in that customers room
than the others (after some adjustment during their installation).

/kc

> 
> 
> David A. Grant,  V.P. Cluster Technologies
> GSH Intelligent Integrated Systems
> 95 Fairmount St. Fitchburg Ma 01450
> Phone 603.898.9717       Fax 603.898.9719
> Email: davidg at gshiis.com      Web: www.gshiis.com
> "Providing High Performance Computing Solutions for Over a Decade"
> ----- Original Message -----
> From: "Velocet" <mathboy at velocet.ca>
> To: "Jim Lux" <James.P.Lux at jpl.nasa.gov>; <rutile at fixy.org>;
> <bcrl at kvack.org>
> Cc: <beowulf at beowulf.org>
> Sent: Tuesday, March 06, 2001 2:49 PM
> Subject: Re: high physical density cluster design -structural...
> 
> 
> > On Tue, Mar 06, 2001 at 11:23:37AM -0800, Jim Lux's all...
> > > Rather than the copper pipe and fittings (which isn't very structural,
> and
> > > will be a pretty significant problem as it gets bigger), you might want
> to
> > > look at some alternatives:
> >
> > Ya, were convening with a few people who've done some work with metal
> > as well as piping this week to go over a few other cheap options. The
> > cluster will be 48 to 64 nodes depending on pricing of other materials,
> > network switches, etc. 48 nodes will be 2 stacks of 24, and at 1U per,
> > thats only 3.5' tall. So we dont need something thats bombproof,
> > just sturdy.
> >
> > > 1) UniStrut (available in aluminum and in galv steel) is much stronger,
> has
> > > nice 90 degree connectors, etc.  There are a variety of similar products
> > > made from aluminum extrusions of one kind or another with longitudinal
> slots
> > > that make very nice rigid boxes. You assemble it with captive nuts and
> > > bolts. The best thing about these products is that they are rectangular,
> not
> > > round, which makes attaching stuff much easier.
> >
> > Hmm, this stuff looks really great - and they seem to be somewhat local
> > to me. :) Looks like it might not be that cheap however, even if it is
> > 'cheap' for industrial applications. Wonder if I can find prices online
> > somewhere here...
> >
> > > 2) Speedrail - a brand of cast aluminum fittings that works with
> aluminum
> > > tubing to make structures, etc. (and hand and safety railings...)
> There
> > > are other brands, as well.  There are versions for 2" and 1" tubing, at
> > > least.  The tubing fits into the socket on the fitting, and you tighten
> set
> > > screws to hold it together.  (Or you can epoxy it....).  For a given $$,
> the
> > > aluminum tubing will be much stronger and more rigid than the copper
> tubing.
> > >
> > >
> > > As far as design guidelines go,  a 0.6 g side load, or so, would be an
> > > appropriate number.  For instance, you should build it strong enough so
> that
> > > you can (gently) tip it over on it's side and not have it fall apart
> during
> > > the move.  In even a small earthquake, poorly braced sheet metal racks
> > > loaded with many pounds of equipment just crumple.  Especially on less
> > > expensive racking, a lot of the strength depends on the sides not
> buckling,
> > > and once it bends even a little bit, it just caves in.
> > >
> > > After all, some day, you WILL have to move the rack a bit, even if only
> a
> > > few feet to let them take up the tile underneath it.
> >
> > True. I dont have a scale, but the board with CPU and ram is about 1.5 or
> > 2lbs, and the power supply is 2-3lbs. That adds up with 48 or 64 odd
> > boards. (Need to figure out if I am going to double up the mainboards
> > per powersupply, would save alot of weight).
> >
> > Thanks for the pointers!
> >
> > /kc
> >
> > >
> > > >> >
> > > >> > Problem 1
> > > >> > """""""""
> > > >> > The problem is in the diagram above, the upside down board has
> another
> > > >> board
> > > >> > .5" above it - are these two boards going to leak RF like mad and
> > > >> interefere
> > > >> > with eachothers' operations? I assume there's not much to do there
> but
> > > to
> > > >> put
> > > >> > a layer of grounded (to the cabinet) metal in between.  This will
> drive
> > > up
> > > >> the
> > > >> > cabinet construction costs. I'd rather avoid this if possible.
> > > >> >
> > > >> > Our original construction was going to be copper pipe and
> plexiglass
> > > >> sheeting,
> > > >> > but we're not sure that this will be viable for something that
> could be
> > > >> rather
> > > >> > tall in our future revisions of our model. Then again, copper pipe
> can
> > > be
> > > >> > bolted to our (cement) ceiling and floor for support.
> > > >> >
> > >
> > >
> >
> > --
> > Ken Chase, math at velocet.ca  *  Velocet Communications Inc.  *  Toronto,
> CANADA
> >
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org
> > To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Ken Chase, math at velocet.ca  *  Velocet Communications Inc.  *  Toronto, CANADA 