[Beowulf] Re: Beowulf of bare motherboards
James.P.Lux at jpl.nasa.gov
Tue Sep 28 10:00:57 PDT 2004
At 11:16 PM 9/27/2004 -0400, Glen Gardner wrote:
>I've gone on a bit of a rant here, but I've done this, so I thought I
>might share some of what I have learned.
>You can see some pictures of what I'm discussing below in the mini cluster
>links on my web site at:
>I have built a cluster on standoffs, and it has worked quite well. While I
>am a relative newbie to Beowulf, I'm not new to computing and electronics.
>I have roughly 40 years experience in electronics.
>Hand fabricating enclosures is a lot of work. It also means you hand
>fabricate a lot (or all) of the witing and power distribution circuitry. I
>can tell you from experience that you will get very tired of drilling
>hundreds of holes, and cutting hundreds of metal components by hand. This
>needs to be a labor of love, because it is a ton of work.
Boy, will I second that... It's one thing to drill a half dozen holes to
hold one motherboard, it's entirely another to drill a hundred! And mark,
and centerpunch, and deburr, and on and on. If you look in the yellow
pages under "sheet metal fabrication" you might find a local vendor who
could do a lot of the grunt work very cheaply (things like shearing the
metal, punching all the holes in exactly the same place, etc.) Don't
discard the idea of contracting out until you've called a couple places.
>You can build a cluster in this manner, but you have several problems to
>consider. I've listed just a few of the "biggies" for your consideration.
>1) thermal issues
>Mounting power hungry boards close to each other can lead to thermal
>issues due to heat radiating and/or convecting from one board to the next.
>other due to radiation and convection of heat. You will still need to
>provide adequate air circulation (this means fans, no matter what you do)
However, a big advantage is that you can use a few BIG fans, which will
move a lot more air, more quietly, than a zillion little fans. Fan
efficiency goes way up as the diameter increases.
>Also, motherboards have internal copper plane layers for grounds and power
>which act as rf shielding and to help spread heat more evenly across the
>board. You will notice that the mounting holes for most motherboards have
>grounded pads around them , and there are sometimes many tiny metal plated
>holes around the mounting hole pad. These are "vias" which are connected
>to the intenral plane layers thermally ( and to ground electrically). IF
>these are mechanically secured to the chassis with a metal standoff, you
>not only benefit from an electrical ground to the chassis (allowing the
>chassis to act as a faraday shield) , you get a thermal path to the
>chassis to help dump some of the excess heat from the board. This is no
>accident, this is a design feature of the board, and a common trick used
>by engineers to provide for dumping heat from the board itself. For this
>reason, and for mechanical reasons, I'd suggest using ALL of the provided
>mounting holes. Believe it or not, it costs money to add a hole to a
>board. (Yes a hole is a design feature), so you can bet the engineer had a
>good reason for putting "extra" holes in a board beyond those required for
>a standard motherboard footprint.
Of course, most commodity mfrs mount the boards with plastic standoffs. You
might actually worsen the EMI performance if you add other ground
points. On the other hand, grounded metal planes between the boards can
only be a "good thing".
>2) RFI issues
>The biggest RFI issue is interference to radio and TV reception. Believe
>it or not there are FCC regulations governing unwanted radio emissions,
>and if your machine is
>making a real racket on radio frequencies, you could concievably have a
>lot of headaches trying to fix the problem after the FCC shows up and asks
>you to shut down.
And this can be very, very difficult to fix. Distance from the victim
receiver is your friend!
>RFI from board to board is something I have yet to encounter in moderm
>pc's. I have encountered it many times in radio circuitry. The solution is
>always the same.
>Use shielding, decouple the power supply properly, and make sure all the
>hardware and shielding is in place, and use a tight fitting metal enclosure.
>For us , that means mounting each motherboard on a metal plate, and making
>sure that each board is grounded to that plate. In turn , the plates are
>securely connected to each other and the chassis. For the enclosure, it
>should be metal, tight fitting and not having large holes. Power wiring
>should be dressed carefully, and long power leads should be twisted and
>may require common mode chokes, etc.
There's a fair amount of information on designing for EMI on the web. As
Glen writes, the basic idea is "put it in a box with the minimum number of
wires penetrating the box, and shield/filter the heck out of those
penetrations". If you can feed in only AC power, and have some sort of
wireless scheme for keyboard, video, network, you're way ahead.
Before you laugh at the idea of using wireless in a shielded box, consider
Holes with a perimeter on the order of a wavelength will pass signals of
that wavelength or shorter.
The EMI you're worried about in a PC is largely below 100-200 MHz
(wavelength on the order of a meter), so a pretty big hole in the shield is
ok (as long as it doesn't have anything conductive penetrating it (like a
wire or conductive bolt). Say you've got a 4 inch diameter hole (for,
e.g., cooling air). Such a hole won't leak very much at 50-100 MHz
(wavelength 3-6 meters). The signals from a wireless LAN or bluetooth
adapter working at 2.4 GHz (wavelength on the order of 12cm) will go right
through that hole with little attenuation.
Watch out for slots where two panels join. They are notorious for leakage,
because they have a long perimeter, and act as a slot radiator. Babinet's
principal says that complementary configurations are basically identical:
it can either be a void in a solid sheet (a slot), or a solid thing in a
void (a wire).
James Lux, P.E.
Spacecraft Radio Frequency Subsystems
Flight Telecommunications Systems
Jet Propulsion Laboratory, Mail Stop 161-213
4800 Oak Grove Drive
Pasadena CA 91109
More information about the Beowulf