[Beowulf] Re: Beowulf of bare motherboards

[Glen Gardner]

I've gone on a bit of a rant here, but I've done this, so I thought I 
might share some of what I have learned.
You can see some pictures of what I'm discussing below in the mini 
cluster links on my web site at:

http://members.bellatlantic.net/~vze24qhw/index.html


I have built a cluster on standoffs, and it has worked quite well. While 
I am a relative newbie to Beowulf, I'm not new to computing and 
electronics. I have roughly 40 years experience in electronics.

Hand fabricating enclosures is a lot of work. It also means you hand 
fabricate a lot (or all) of the witing and power distribution circuitry. 
I can tell you from experience that you will get very tired of drilling 
hundreds of holes, and cutting hundreds of metal components by hand. 
This needs to be a labor of love, because it is a ton of work.

You can build a cluster in this manner, but you have several problems to 
consider. I've listed just a few of the "biggies" for your consideration.

1) thermal issues
Mounting power hungry boards close to each other can lead to thermal 
issues due to heat radiating and/or convecting from one board to the next.
If boards are very close to each other, you are likely to need to add 
additional fans to move air abundantly around the boards and cpu's to 
keep the boards from cooking each other. One way to improve this is to 
mount each board on a metal plate and stack the plates on metal 
standoffs. Having metal plates between the boards will provide 
significant thermal shielding and alleviate the tendency of the boards 
to "cook" each other due to radiation and convection of heat. You will 
still need to provide adequate air circulation (this means fans, no 
matter what you do)

Also, motherboards have internal copper plane layers for grounds and 
power which act as rf shielding and to help spread heat more evenly 
across the board. You will notice that the mounting holes for most 
motherboards have grounded pads around them , and there are sometimes 
many tiny metal plated holes around the mounting hole pad. These are 
"vias" which are connected to the intenral plane layers thermally ( and 
to ground electrically). IF these are mechanically secured to the 
chassis with a metal standoff, you not only benefit from an electrical 
ground to the chassis (allowing the chassis to act as a faraday shield) 
, you get a thermal path to the chassis to help dump some of the excess 
heat from the board. This is no accident, this is a design feature of 
the board, and a common trick used by engineers to provide for dumping 
heat from the board itself. For this reason, and for mechanical reasons, 
I'd suggest using ALL of the provided mounting holes. Believe it or not, 
it costs money to add a hole to a board. (Yes a hole is a design 
feature), so you can bet the engineer had a good reason for putting 
"extra" holes in a board beyond those required for a standard 
motherboard footprint.


2) RFI issues
The biggest RFI issue is interference to radio and TV reception. Believe 
it or not there are FCC regulations governing unwanted radio emissions, 
and if your machine is
making a real racket on radio frequencies, you could concievably have a 
lot of headaches trying to fix the problem after the FCC shows up and 
asks you to shut down.

RFI from board to board is something I have yet to encounter in moderm 
pc's. I have encountered it many times in radio circuitry. The solution 
is always the same.
Use shielding, decouple the power supply properly, and make sure all the 
hardware and shielding is in place, and use a tight fitting metal 
enclosure.

For us , that means mounting each motherboard on a metal plate, and 
making sure that each board is grounded to that plate. In turn , the 
plates are securely connected to each other and the chassis. For the 
enclosure, it should be metal, tight fitting and not having large holes. 
Power wiring should be dressed carefully, and long power leads should be 
twisted and may require common mode chokes, etc.

Board to board interference
I can imagine a situation where motherboards having a high power demand 
, in close proximity to each other might cause interference to each 
other. But I think that mounting each board on a single metal plate 
securely , with conductive standoffs for good grounding, would be more 
than adequate to prevent motherboards from talking to each other in 
virtually every situation. I'm presently stacking 3 VIA ( @ 15 W each) 
motherboards on one metal plate, and I have 4 stacks of 3 motherboards. 
They do not interfere with each other. A higher power cpu might cause 
interference in my setup, but mounting each board on its own metal 
"shielding" plate in the stack would alliveiate this situation.


3) power distribution issues (this part is only relevant if you are 
using as odd a power supply setup as I am)
This is a problem I am presently wrestling with at the nusiance level. I 
know the cure and am in the process of implimenting part of it now.
My computational nodes are in two racks of 6 nodes , each 6 node rach 
has two stacks of 3 boards, with a metal plate on top , and a metal 
plate on the bottom.
On the mettal plate is a DC-DC converter board which converts 12VDC to 
the required ATX voltages used by the mini itx boards. All of the DC-DC 
converters are fed from a single 12VDC regulated power supply at a power 
distribution panel built into the top of each 6 node rack.

The issue I have encountered has caused two problems; First, when 
powering up "cold", I occasionally have a node fail to boot correctly. 
Also, I've observed that once in a while, at turnon, the NFS server hard 
drive head will seek home , hard with an audible click, suggesting that 
it is "unhappy" with something.
It turns out that with many nodes drawing current at turnon, across a 
long power distribution, the nodes (and hard drive) are attemting to 
start before the 12VDC power has stablilized. After almost a year of 
operation, the mini cluster has devoured two full size ide hard drives 
used for it's NFS mount. I also have 12 IBM microdrives in the nodes, 
and have only lost one (after it was abused thermally during some 
difficult testing). Interestingly enough the high failure rate IDE 
drives are on the 12V supply, and the lower failure rate microdrives are 
on a conditined 5V supply. It turns out that the 12VDC regulated supply 
is not coming up fast enough at turnon, and as a result the hardware is 
tending to glitch occasionally due to a "noisy" power supply.

My solution for this is going to involve the following: Replace the IDE 
hard drive with a laptop drive that will operate on the much cleaner 5V 
supply , and to save power and reduce inrush currents at turnon. Also 
I'd like to sequence the turnon of the nodes such that the inrush 
currents at turnon are lower, allowing the 12V supply to come up to a 
stable 12VDC a bit faster.

At this point in time , my biggest concern is the occasional loss of the 
ide drives, so I am focusing on the hard drive issue, but I have in mind 
building a power control sequencer to bring up the nodes in a safe and 
sane manner. (more on this later, if I actually do it).


4) mechanical integrity
Your design needs to be mechanically sound. Think things through and do 
a good job. If you have never designed anything this large before you 
might be wise to allow a lot of extra time and some extra materials so 
you can get through the learning curve a bit easier (plan on mistakes).

The mini cluster racks were designed to be cheap and effective. 
(Emphasis on cheap) A happy consequence of the design is that it is 
compact, lightweight and strong. If I had it to do again , I'd have 
built in a little more servicability at a slightly higher cost, because 
the replacement of a node requires dismantling the racks completely.



Good luck;

Glen


Jack Wathey wrote:

>
>
> On Mon, 27 Sep 2004, Andrew Piskorski wrote:
>
>>> http://www.abo.fi/~physcomp/cluster/celeron.html
>>
>>
>> I recently bought a bunch of old ECS P4VXMS motherboards cheap on
>> Ebay, in order to build my own personal home cluster for testing and
>> experimentation. I plan to mount them on metal shelves somewhat like
>> the Celeron cluster shown above. However, that raises the following
>> questions:
>>
>> Normally, motherboards are mounted to the metal case using metal
>> standoffs, which touch the motherboard ONLY at the designed mounting
>> holes. Is this in fact necessary? Desirable?
>>
>> Is there any reason the standoffs need to be conductive?
>>
>> Would it be ok to instead simply screw the motherboard down to a flat
>> piece of wood or plastic? Or to wood or plastic strips which touch
>> all along much of the bottom of the motherboard?
>>
>
> Andrew,
>
> About a year ago I built a cluster much as you describe (bare 
> motherboards on steel shelves), but with the motherboards mounted to 
> aluminum sheets instead of wood or plastic. I used nylon standoffs 
> (Mouser electronics part #561-A0250; www.mouser.com), and everything 
> worked fine. The motherboards are properly grounded by virtue of the 
> ground wires in the connector to the power supply, as are the aluminum 
> sheets, but the standoffs are nonconducting. The standoffs just snap 
> into the aluminum sheets (which are 1/16 inch thick) and snap into the 
> motherboard holes. There are no threads and no nuts involved.
>
> Ken Chase (math at velocet.ca) has posted to this list, a year or 2 ago, 
> about his experiments with packing motherboards close together in a 
> plexiglas enclosure with no shielding between them. I believe he 
> concluded that this was a bad idea, because radio-frequency interference
> between the boards caused some of them to fail to boot. Also large 
> quantities of wood or plastic near a motherboard is a bad idea because 
> of the risk of fire.
>
> Unfortunately I don't have a website describing my cluster, but, if 
> you're interested, I could send you more details. One final word of 
> warning: building a cluster like this (bare motherboards in a 
> customized enclosure) is LOT more work than you probably expect. It 
> was for me, anyway. I didn't have much choice, because of space 
> constraints, but you might give
> serious thought to the more conventional approach.
>
> Hope this helps,
>
> Jack
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
Glen E. Gardner, Jr.
AA8C
AMSAT MEMBER 10593



http://members.bellatlantic.net/~vze24qhw/index.html