[Beowulf] Re: Beowulf of bare motherboards

[Jim Lux]

At 11:16 PM 9/27/2004 -0400, Glen Gardner wrote:
>I've gone on a bit of a rant here, but I've done this, so I thought I 
>might share some of what I have learned.
>You can see some pictures of what I'm discussing below in the mini cluster 
>links on my web site at:
>
>http://members.bellatlantic.net/~vze24qhw/index.html
>
>
>I have built a cluster on standoffs, and it has worked quite well. While I 
>am a relative newbie to Beowulf, I'm not new to computing and electronics. 
>I have roughly 40 years experience in electronics.
>
>Hand fabricating enclosures is a lot of work. It also means you hand 
>fabricate a lot (or all) of the witing and power distribution circuitry. I 
>can tell you from experience that you will get very tired of drilling 
>hundreds of holes, and cutting hundreds of metal components by hand. This 
>needs to be a labor of love, because it is a ton of work.

Boy, will I second that... It's one thing to drill a half dozen holes to 
hold one motherboard, it's entirely another to drill a hundred! And mark, 
and centerpunch, and deburr, and on and on.  If you look in the yellow 
pages under "sheet metal fabrication" you might find a local vendor who 
could do a lot of the grunt work very cheaply (things like shearing the 
metal, punching all the holes in exactly the same place, etc.)  Don't 
discard the idea of contracting out until you've called a couple places.




>You can build a cluster in this manner, but you have several problems to 
>consider. I've listed just a few of the "biggies" for your consideration.
>
>1) thermal issues
>Mounting power hungry boards close to each other can lead to thermal 
>issues due to heat radiating and/or convecting from one board to the next.
<snip>
>other due to radiation and convection of heat. You will still need to 
>provide adequate air circulation (this means fans, no matter what you do)


However, a big advantage is that you can use a few BIG fans, which will 
move a lot more air, more quietly, than a zillion little fans.  Fan 
efficiency goes way up as the diameter increases.


>Also, motherboards have internal copper plane layers for grounds and power 
>which act as rf shielding and to help spread heat more evenly across the 
>board. You will notice that the mounting holes for most motherboards have 
>grounded pads around them , and there are sometimes many tiny metal plated 
>holes around the mounting hole pad. These are "vias" which are connected 
>to the intenral plane layers thermally ( and to ground electrically). IF 
>these are mechanically secured to the chassis with a metal standoff, you 
>not only benefit from an electrical ground to the chassis (allowing the 
>chassis to act as a faraday shield) , you get a thermal path to the 
>chassis to help dump some of the excess heat from the board. This is no 
>accident, this is a design feature of the board, and a common trick used 
>by engineers to provide for dumping heat from the board itself. For this 
>reason, and for mechanical reasons, I'd suggest using ALL of the provided 
>mounting holes. Believe it or not, it costs money to add a hole to a 
>board. (Yes a hole is a design feature), so you can bet the engineer had a 
>good reason for putting "extra" holes in a board beyond those required for 
>a standard motherboard footprint.

Of course, most commodity mfrs mount the boards with plastic standoffs. You 
might actually worsen the EMI performance if you add other ground 
points.  On the other hand, grounded metal planes between the boards can 
only be a "good thing".



>2) RFI issues
>The biggest RFI issue is interference to radio and TV reception. Believe 
>it or not there are FCC regulations governing unwanted radio emissions, 
>and if your machine is
>making a real racket on radio frequencies, you could concievably have a 
>lot of headaches trying to fix the problem after the FCC shows up and asks 
>you to shut down.

And this can be very, very difficult to fix.  Distance from the victim 
receiver is your friend!


>RFI from board to board is something I have yet to encounter in moderm 
>pc's. I have encountered it many times in radio circuitry. The solution is 
>always the same.
>Use shielding, decouple the power supply properly, and make sure all the 
>hardware and shielding is in place, and use a tight fitting metal enclosure.
>
>For us , that means mounting each motherboard on a metal plate, and making 
>sure that each board is grounded to that plate. In turn , the plates are 
>securely connected to each other and the chassis. For the enclosure, it 
>should be metal, tight fitting and not having large holes. Power wiring 
>should be dressed carefully, and long power leads should be twisted and 
>may require common mode chokes, etc.
>

There's a fair amount of information on designing for EMI on the web.  As 
Glen writes, the basic idea is "put it in a box with the minimum number of 
wires penetrating the box, and shield/filter the heck out of those 
penetrations".  If you can feed in only AC power, and have some sort of 
wireless scheme for keyboard, video, network, you're way ahead.

Before you laugh at the idea of using wireless in a shielded box, consider 
this...

Holes with a perimeter on the order of a wavelength will pass signals of 
that wavelength or shorter.

The EMI you're worried about in a PC is largely below 100-200 MHz 
(wavelength on the order of a meter), so a pretty big hole in the shield is 
ok (as long as it doesn't have anything conductive penetrating it (like a 
wire or conductive bolt).  Say you've got a 4 inch diameter hole (for, 
e.g., cooling air). Such a hole won't leak very much at 50-100 MHz 
(wavelength 3-6 meters). The signals from a wireless LAN or bluetooth 
adapter working at 2.4 GHz (wavelength on the order of 12cm) will go right 
through that hole with little attenuation.

Watch out for slots where two panels join.  They are notorious for leakage, 
because they have a long perimeter, and act as a slot radiator.  Babinet's 
principal says that complementary configurations are basically identical: 
it can either be a void in a solid sheet (a slot), or a solid thing in a 
void (a wire).




James Lux, P.E.
Spacecraft Radio Frequency Subsystems
Flight Telecommunications Systems
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
tel: (818)354-2075
fax: (818)393-6875