[Beowulf] A Cluster of Motherboard.

[Jim Lux]

At 06:38 PM 11/10/2005, Glen Gardner wrote:
>I think the advantage to stacking bare boards is the potential for
>greater density in terms of number of cpus, and a potential for better
>thermal management.  There is nothing to stop one from forcing massive
>amounts of air through closely spaced boards to get the density up so
>that large clusters take up a lot less valuable floor space.

Uhh.. thermal management for large systems can be a lot more complex than 
just pushing bucketloads of air through.  Ask the folks designing blade 
systems.

So, you're right, there's a *potential* for better thermal management, but 
there's a non zero cost to realize that potential.  At least with stock 
cases, someone has presumably done the thermal design.


>The cost of an aluminum plate and standoffs is trivial compared to a
>case, and once you have made one plate all the others can be readily
>mass produced using a template and a hand drill.

The material cost may be low, but you're not going to be out making 90 odd 
aluminum plates with a hand drill (I assume you're talking an electric 
drill, not an "eggbeater"), unless you're not very picky about tolerances 
and fit.


>I can fabricate a chassis for a mini itx cluster from aluminum plate and
>aluminum angle stock with simple hand tools in about 2 days.... and I
>could do it in hours if I owned a drill press and cutoff saw.
>
>The main point is, it can be done cheaply and with simple tools.

Yes, but if you pay yourself any sort of reasonable rate, it's not 
necessarily cheaper, especially when you consider assembly time.  For a 
hobby cluster, sure, you can count your time as free, and then, you can 
make the tradeoff for enjoyable free time vs writing a bigger check.

2 days of hiring someone basically competent is probably about $500.
(We used to hire moderately skilled carpenters to build sets or special 
effects technicians at a cost of about $25/hr, back in 1995.  Call it 
$250/day.)

Mind you, though, you're not going to be building any 100 node cluster with 
a 2 day labor investment.  Just the time to pick up, walk across the room, 
and put down the 100 aluminum plates would be more than that.



>There is nothing unprofessional about such setups... they are very
>professional and offer layout flexibility that opens up new
>possibilities.
>
>My present setup has vertically arranged mini itx boards bolted to a
>rack panel.  One could easily stack 18 mini itx form factor p4
>motherboards with power supplies and hard drives behind a 12 inch high
>rack panel, and have 7 of those arrays in a single equipment rack for a
>total of 126 nodes in a single equipment rack.  with the vertical P4
>boards you would have to force cooling air through the rack, but it will
>work fine.
>
>If you use dual cpu mini itx boards, that gives you 252 cpu's and a much
>easier cooling solution.... I am sure that one can stack them in an even
>denser fashion if they tried....

Hmmm.. lets say 100 Watts each.  250 cpus is then about 25kW.  Say you have 
cooling air at 25C and you will allow a 10C rise.  The specific heat for 
air is about 1 kJ/kg*K and the density is about 1 kg/m3.  25kW is 25kJ/sec, 
so we need about 2.4 kg/sec of air (for the 10 degree rise) or 2.4 m3/sec.

This is about like a couple or three electric clothes dryers, by the way.

Let's assume your pushing that air in through the entire front face of the 
rack which is roughly 18"x72" (or, keeping it metric, 0.45x1.8 m = .81 m2). 
The air flow speed through the rack is about 3 m/sec.

Most folks actually buy fans in CFM and measure air flow in LFPM.. 35 cf/m3 
so, about 5000 cfm.  3m/sec is about 600 LFPM, a reasonable speed.

Whipping out the W.W. Grainger catalog.. there's a nice 10" diameter axial 
fan that pushes 665 CFM.  Of course, that's at some negligble pressure 
drop, so you probably can only figure about 300 CFM in practice.  So, you 
need to buy 16 of them, they'll just fit on the face of your rack.
They're $60 each so that will set you back another kilobuck.
And, of course, you'll need to make sure you package the fans right, and 
find a place for that 5000CFM of air to come from and go to. (A couple big 
2x6 foot holes in the wall?)

(rgb quotes 1 ton of A/C per 3500W, so we're looking at a 4 ton air 
conditioner)

>It is all about cpu density vs floor/rack space and thermal management,
>and I am surprised that someone has not come up with a low cost
>commercially available cabinet for beowulf clusters that embodied this
>idea...


Because anyone who can afford the cost of the 100+ motherboards and the 
electricity to run them, probably also has to deal with the cooling issues, 
and the cost of the packaging starts to become insignificant.  (25 kW @ 
0.20/kWh costs you $5/hr just in electricity.  Run that puppy 24/7, and 
it's almost $50k/yr... )


>Glen Gardner
>

bareboards and ad hoc fabrication is fine for a few or a dozen mobos.. When 
you start talking a hundred, there's a lot more to it, and casual design 
MIGHT work, but if you're investing hundreds of kilobucks, odds are you 
want SURE to work.