[Beowulf] A Cluster of Motherboard.

Mon Nov 14 05:27:44 PST 2005

On Thu, 10 Nov 2005, Glen Gardner wrote:

Dear Glen,

It would be really lovely if you would post some photos.  A picture or
three is worth a lot of text here, and although google reveals a FEW
"recipe" websites they are so widely disparate in approach that there
isn't as much advantage as one would like (I did a bit of a search while
researching the answer to this question for the original poster).

The other thing that I didn't see addressed this time around is the
power supply issue.  In previous discussions and from what I've seen of
commercial designs (e.g. rackable) one major advantage to this
particular kind of DIY cluster is the ability to use big power supplies
to run many little boards, because the larger supplies are relatively
more efficient.  However, I also recall there were "issues" -- getting
the right wiring harness to be able to power multiple motherboards,
getting a large PS in the first place, locating PS relative to
motherboards so that you can get the power where it belongs.  Designs
with one PS/mobo vs one PS per "tray" vs one PS per "shelf" -- CBA and
engineering advantage comparisons -- seem to be warranted.

I also have been following the tape vs screw spacer part of the thread
with great interest.  What one might REALLY want to see (in regular
cases as well) is a simple motherboard clamp.  For example, the little
locking spring-loaded clamp they have in plastic folders for holding
down papers -- slide the motherboard into place, clamp.  No screws, no
holes, three seconds in three seconds out.

[Aside -- This is all part of design flaws that date back to the
original IBM PC.  The screw-in card bus is just plain evil -- it is SO
easy to design a screwless thumb-able card lock -- snap in the board,
thumb down the lock.  Certain Dell systems I've gotten over the years
have even had them.  They can do this for the CPU -- ZIF sockets were a
great blessing -- so why not everything else?

I'd REALLY like to see PC components in general reengineered so one does
not need a screwdriver to assemble them -- snap in, clamp down, plug
in.]

Cheap spring clamps are probably available at pennies per, and if one
uses steel plates instead of aluminum one can "attach" them with equally
cheap strip magnets (or one could just glue them onto aluminum stock).
That saves the time consuming drilling and gets you around the problems
I see with double sided mounting tape -- removal is likely to be messy
and possibly warranty-voiding, no ground (screws and spacers basically
ground the motherboard to the chassis in addition to providing
mechanical strength), time and force required to remove clean OR messy,
the difficulty of finding tape that is likely to not be a fire hazard or
melting hazard and that won't dry out, shrink, and pull free under the
constant drying-oven heat of its operational environment.

The one thing you might need to add is "bumper" supports under the card
bus and memory bus.  Cheap motherboards tend to flex when you push a
card into a socket and having a support here prevents the motherboard
bottom from contacting the plate and maybe shorting out (if you are as
lazy/sloppy as I am and don't always remove the plug and wait for the PS
capacitor to discharge before inserting a card:-).  Also, cheap mobos
are CHEAP, and I don't LIKE to flex them in case it cracks an internal
trace that might be marginal but function if unflexed...

    rgb

> I think the advantage to stacking bare boards is the potential for
> greater density in terms of number of cpus, and a potential for better
> thermal management.  There is nothing to stop one from forcing massive
> amounts of air through closely spaced boards to get the density up so
> that large clusters take up a lot less valuable floor space.
>
> The cost of an aluminum plate and standoffs is trivial compared to a
> case, and once you have made one plate all the others can be readily
> mass produced using a template and a hand drill.
>
> I can fabricate a chassis for a mini itx cluster from aluminum plate and
> aluminum angle stock with simple hand tools in about 2 days.... and I
> could do it in hours if I owned a drill press and cutoff saw.
>
> The main point is, it can be done cheaply and with simple tools.
>
> There is nothing unprofessional about such setups... they are very
> professional and offer layout flexibility that opens up new
> possibilities.
>
> My present setup has vertically arranged mini itx boards bolted to a
> rack panel.  One could easily stack 18 mini itx form factor p4
> motherboards with power supplies and hard drives behind a 12 inch high
> rack panel, and have 7 of those arrays in a single equipment rack for a
> total of 126 nodes in a single equipment rack.  with the vertical P4
> boards you would have to force cooling air through the rack, but it will
> work fine.
>
> If you use dual cpu mini itx boards, that gives you 252 cpu's and a much
> easier cooling solution.... I am sure that one can stack them in an even
> denser fashion if they tried....
>
> It is all about cpu density vs floor/rack space and thermal management,
> and I am surprised that someone has not come up with a low cost
> commercially available cabinet for beowulf clusters that embodied this
> idea...
>
>
> Glen Gardner
>
>
>
>
> On Thu, 2005-11-10 at 10:27 -0800, Jim Lux wrote:
>> At 08:54 AM 11/10/2005, Andrew Piskorski wrote:
>>> On Thu, Nov 10, 2005 at 05:35:03AM -0800, Jim Lux wrote:
>>>
>>>> However, lots of people have successfully built clusters from
>>> stacks of
>>>> mobos.  I think the biggest one (in terms of # of nodes) is the
>>> one with a
>>>> dozen or so Via mini-ATX boards.  I don't know that I've seen any
>>> bare
>>>> bones clusters with more than 20 nodes.
>>>
>>> Oh, I know of at least 3 bare-board clusters much larger than that
>>> (and some smaller):
>>
>> Excellent examples.. (what comes from making an off the cuff comment
>> before having my first cup of coffee in the morning)..
>>
>>>   http://jessen.ch/ammonite/
>>
>> Telling comment:
>>
>> "Perhaps the most helpful thing I could say is to urge you to consider
>> building a conventional cluster (shelves of COTS midtower cases or
>> racks of 1U pizza boxes) instead of something like ammonite. The
>> ammonite design has some advantages (high cpu density, better
>> ventilation and lower delta-T, for example), but designing and
>> building it was a colossal time sink. ...
>> There were many little things that had to be custom made or modified,
>> no one of which was a big deal, but all of which together were a very
>> big deal. " (ellipses mine, JL)
>>
>>
>>
>>>   http://joule.bu.edu/~hazen/LinuxCluster/
>>
>>
>> "Instead of a flat plate we use a custom-made aluminum box with
>> punched holes and welded corners made by a local sheet-metal
>> house. ..."
>> Dividing out their $4500 hardware cost, they spent just under
>> $100/mobo for packaging (their box was probably about half that).
>> Note well, they don't mention labor costs.. If you go custom box, you
>> want to look closely at boxes designed for mass production: sliding
>> Tab A into Slot B is a heck of a lot faster than fumbling for 6-32
>> screws.
>>
>>>   http://krone.physik.unizh.ch/~stadel/zBox/
>>
>> "
>> We greatly acknowledge the aid of the Physics Mechanical Workshop at
>> the University of Zurich for: 1) turning the "napkin-sketch" into a
>> proper CAD/CAM design of the machine; 2) providing numerous
>> suggestions which improved the detailed design; 3) providing a
>> gigantic room for the construction of the boards; 4) and, well,
>> building the thing!
>> "
>>
>> ---
>> In any case, ALL of these bigger systems had fairly custom designed
>> (read: not cheap) packaging hardware.  They are pretty nifty looking.
>>
>>
>> They also raise some "serviceability" questions too...
>>
>> Jim
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu