[Beowulf] Re: Beowulf of bare motherboards
Glen.Gardner at verizon.net
Mon Sep 27 20:16:36 PDT 2004
I've gone on a bit of a rant here, but I've done this, so I thought I
might share some of what I have learned.
You can see some pictures of what I'm discussing below in the mini
cluster links on my web site at:
I have built a cluster on standoffs, and it has worked quite well. While
I am a relative newbie to Beowulf, I'm not new to computing and
electronics. I have roughly 40 years experience in electronics.
Hand fabricating enclosures is a lot of work. It also means you hand
fabricate a lot (or all) of the witing and power distribution circuitry.
I can tell you from experience that you will get very tired of drilling
hundreds of holes, and cutting hundreds of metal components by hand.
This needs to be a labor of love, because it is a ton of work.
You can build a cluster in this manner, but you have several problems to
consider. I've listed just a few of the "biggies" for your consideration.
1) thermal issues
Mounting power hungry boards close to each other can lead to thermal
issues due to heat radiating and/or convecting from one board to the next.
If boards are very close to each other, you are likely to need to add
additional fans to move air abundantly around the boards and cpu's to
keep the boards from cooking each other. One way to improve this is to
mount each board on a metal plate and stack the plates on metal
standoffs. Having metal plates between the boards will provide
significant thermal shielding and alleviate the tendency of the boards
to "cook" each other due to radiation and convection of heat. You will
still need to provide adequate air circulation (this means fans, no
matter what you do)
Also, motherboards have internal copper plane layers for grounds and
power which act as rf shielding and to help spread heat more evenly
across the board. You will notice that the mounting holes for most
motherboards have grounded pads around them , and there are sometimes
many tiny metal plated holes around the mounting hole pad. These are
"vias" which are connected to the intenral plane layers thermally ( and
to ground electrically). IF these are mechanically secured to the
chassis with a metal standoff, you not only benefit from an electrical
ground to the chassis (allowing the chassis to act as a faraday shield)
, you get a thermal path to the chassis to help dump some of the excess
heat from the board. This is no accident, this is a design feature of
the board, and a common trick used by engineers to provide for dumping
heat from the board itself. For this reason, and for mechanical reasons,
I'd suggest using ALL of the provided mounting holes. Believe it or not,
it costs money to add a hole to a board. (Yes a hole is a design
feature), so you can bet the engineer had a good reason for putting
"extra" holes in a board beyond those required for a standard
2) RFI issues
The biggest RFI issue is interference to radio and TV reception. Believe
it or not there are FCC regulations governing unwanted radio emissions,
and if your machine is
making a real racket on radio frequencies, you could concievably have a
lot of headaches trying to fix the problem after the FCC shows up and
asks you to shut down.
RFI from board to board is something I have yet to encounter in moderm
pc's. I have encountered it many times in radio circuitry. The solution
is always the same.
Use shielding, decouple the power supply properly, and make sure all the
hardware and shielding is in place, and use a tight fitting metal
For us , that means mounting each motherboard on a metal plate, and
making sure that each board is grounded to that plate. In turn , the
plates are securely connected to each other and the chassis. For the
enclosure, it should be metal, tight fitting and not having large holes.
Power wiring should be dressed carefully, and long power leads should be
twisted and may require common mode chokes, etc.
Board to board interference
I can imagine a situation where motherboards having a high power demand
, in close proximity to each other might cause interference to each
other. But I think that mounting each board on a single metal plate
securely , with conductive standoffs for good grounding, would be more
than adequate to prevent motherboards from talking to each other in
virtually every situation. I'm presently stacking 3 VIA ( @ 15 W each)
motherboards on one metal plate, and I have 4 stacks of 3 motherboards.
They do not interfere with each other. A higher power cpu might cause
interference in my setup, but mounting each board on its own metal
"shielding" plate in the stack would alliveiate this situation.
3) power distribution issues (this part is only relevant if you are
using as odd a power supply setup as I am)
This is a problem I am presently wrestling with at the nusiance level. I
know the cure and am in the process of implimenting part of it now.
My computational nodes are in two racks of 6 nodes , each 6 node rach
has two stacks of 3 boards, with a metal plate on top , and a metal
plate on the bottom.
On the mettal plate is a DC-DC converter board which converts 12VDC to
the required ATX voltages used by the mini itx boards. All of the DC-DC
converters are fed from a single 12VDC regulated power supply at a power
distribution panel built into the top of each 6 node rack.
The issue I have encountered has caused two problems; First, when
powering up "cold", I occasionally have a node fail to boot correctly.
Also, I've observed that once in a while, at turnon, the NFS server hard
drive head will seek home , hard with an audible click, suggesting that
it is "unhappy" with something.
It turns out that with many nodes drawing current at turnon, across a
long power distribution, the nodes (and hard drive) are attemting to
start before the 12VDC power has stablilized. After almost a year of
operation, the mini cluster has devoured two full size ide hard drives
used for it's NFS mount. I also have 12 IBM microdrives in the nodes,
and have only lost one (after it was abused thermally during some
difficult testing). Interestingly enough the high failure rate IDE
drives are on the 12V supply, and the lower failure rate microdrives are
on a conditined 5V supply. It turns out that the 12VDC regulated supply
is not coming up fast enough at turnon, and as a result the hardware is
tending to glitch occasionally due to a "noisy" power supply.
My solution for this is going to involve the following: Replace the IDE
hard drive with a laptop drive that will operate on the much cleaner 5V
supply , and to save power and reduce inrush currents at turnon. Also
I'd like to sequence the turnon of the nodes such that the inrush
currents at turnon are lower, allowing the 12V supply to come up to a
stable 12VDC a bit faster.
At this point in time , my biggest concern is the occasional loss of the
ide drives, so I am focusing on the hard drive issue, but I have in mind
building a power control sequencer to bring up the nodes in a safe and
sane manner. (more on this later, if I actually do it).
4) mechanical integrity
Your design needs to be mechanically sound. Think things through and do
a good job. If you have never designed anything this large before you
might be wise to allow a lot of extra time and some extra materials so
you can get through the learning curve a bit easier (plan on mistakes).
The mini cluster racks were designed to be cheap and effective.
(Emphasis on cheap) A happy consequence of the design is that it is
compact, lightweight and strong. If I had it to do again , I'd have
built in a little more servicability at a slightly higher cost, because
the replacement of a node requires dismantling the racks completely.
Jack Wathey wrote:
> On Mon, 27 Sep 2004, Andrew Piskorski wrote:
>> I recently bought a bunch of old ECS P4VXMS motherboards cheap on
>> Ebay, in order to build my own personal home cluster for testing and
>> experimentation. I plan to mount them on metal shelves somewhat like
>> the Celeron cluster shown above. However, that raises the following
>> Normally, motherboards are mounted to the metal case using metal
>> standoffs, which touch the motherboard ONLY at the designed mounting
>> holes. Is this in fact necessary? Desirable?
>> Is there any reason the standoffs need to be conductive?
>> Would it be ok to instead simply screw the motherboard down to a flat
>> piece of wood or plastic? Or to wood or plastic strips which touch
>> all along much of the bottom of the motherboard?
> About a year ago I built a cluster much as you describe (bare
> motherboards on steel shelves), but with the motherboards mounted to
> aluminum sheets instead of wood or plastic. I used nylon standoffs
> (Mouser electronics part #561-A0250; www.mouser.com), and everything
> worked fine. The motherboards are properly grounded by virtue of the
> ground wires in the connector to the power supply, as are the aluminum
> sheets, but the standoffs are nonconducting. The standoffs just snap
> into the aluminum sheets (which are 1/16 inch thick) and snap into the
> motherboard holes. There are no threads and no nuts involved.
> Ken Chase (math at velocet.ca) has posted to this list, a year or 2 ago,
> about his experiments with packing motherboards close together in a
> plexiglas enclosure with no shielding between them. I believe he
> concluded that this was a bad idea, because radio-frequency interference
> between the boards caused some of them to fail to boot. Also large
> quantities of wood or plastic near a motherboard is a bad idea because
> of the risk of fire.
> Unfortunately I don't have a website describing my cluster, but, if
> you're interested, I could send you more details. One final word of
> warning: building a cluster like this (bare motherboards in a
> customized enclosure) is LOT more work than you probably expect. It
> was for me, anyway. I didn't have much choice, because of space
> constraints, but you might give
> serious thought to the more conventional approach.
> Hope this helps,
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit
Glen E. Gardner, Jr.
AMSAT MEMBER 10593
More information about the Beowulf