Physical questions

[Robert G. Brown]

On Tue, 14 Jan 2003, Ken Chase wrote:

> On Tue, Jan 14, 2003 at 12:28:17AM -0400, Sam Daniel's all...
> > Here's a slightly different approach which was successful on a small scale:
> > 
> > http://www.clustercompute.com/
> > 
> > Might not scale up to thousands of nodes, though.... :-)
> 
> or to 10 for that matter. When the middle one fails.
> 
> http://trooper.velocet.net/~math/images/133-3357_IMG.JPG
> 
> is what we did - custom cabinet, with brazeon mount points. Metal shelves.
> 4 nodes per shelf.
> 
> Without metal you get way too much RF. We found that stacking nodes
> really close together vertically with no shielding caused many to not
> boot properly (seemingly the most sensitive time) and many to crash
> randomly (perhaps 10% of nodes every 2 weeks, mysterious crash).
> 
> Moving them to twice the distance between, my guess, reduced the crash
> rate by a factor of 8ish (cube of distance? :)

Hmmm, sounds like some unsurprising mutual inductance in the GHz range,
doesn't it?  Reminds me of the very old days and my first IBM PC.  I
used to work listening to the FM radio in these pre-ogg days when I
didn't have my entire collection of music on my hard disk.  If I was
tuned to anywhere in the vicinity of 100 MHz while the PC was running,
the radio buzzed like crazy.  Don't ask me why -- it had a 4.77 MHz
clock, IIRC, so it must have been harmonics of some sort.  I could
clearly hear the computer "work" -- the buzz was nicely modulated when I
ran a small floating point program, for example.  Made me truly believe
that someone could read your keystrokes off your power supply line or
local system EMF with sufficiently delicate equipment a la
Cryptonomicon...

At a guess you could have also solved your problem by inserting e.g.
aluminum screens with mesh holes on the order of a few mm in between
nodes.  Depending on whether or not you got reflection and self-induced
crash or absorption.

However, as I noted to one person off list, the things that have
prevented me from ever building a cluster like this are:

 a) Liability, fire codes, personal safety, and (as noted) emf.  In many
institutions you'd get flayed alive installing electrical hardware in
non-UL-or-other-safety-underwriter-underwritten configurations.  A small
"lab" or "hobby" configuration maybe -- in a large scale production
facility no no no.  Just establishing predictable airflow cooling and
controlling dust argue against an open large scale design.

 b) Time, beauty, and convenience.  Time because it takes a lot less to
just use OTC parts.  Beauty matters to e.g. granting agencies -- they
are more inclined to think a cluster is a "real" resource if it looks
like a rackmount or even a tower/shelfmount beowulfish computer than if
it looks like my latest science project with everything out naked and
wires exposed.  Convenience (really more time) because if a node fails,
it is encapsulated and can be benched to work on it.  Unbolting a power
supply from a shelf of running units, pulling the motherboard out of a
shelf of running motherboards?  Messing with cards ditto?  Adding a hard
disk to a shelf-mounted system?  Standardization saves time, and cases
are reasonably standardized.

Custom stuff pushes you out of the OTC/OTS sweet spot, into greater
risk, less explored engineering terrain, uglier and less
professional-looking installations, and ultimately can cost time and
money beyond what you expect to save.  I'd avoid them unless you are a
devoted hobbiest and/or have some very specific engineering constraint
that can't be met with OTC parts and configurations.

With that said, I >>have<< had occasional fantasies about doing a
home/hobby shelf of nodes this way.  There my time is "free", my space
is very limited.  OTOH, I burn myself, my family, and my pets alive if
it starts a fire in the middle of the night that gets out of hand too
quickly, or maybe just lose everything I own.  And presumably safe(r)
tower cases that might possibly confine a the heat/fire from an internal
short until a fuse blows are so cheap.  And my nodes accumulate dust
(see "pets") so fast even when closed...

   rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu