[Beowulf] Newbie Question: Racks versus boxes and good rack solutions for commodity hardware

[Robert G. Brown]

On Thu, 11 Dec 2008, arjuna wrote:

> Hello all again:
> 
> I thought I would add a little more background about myself and the intended
> cluster. I am an artist and a computer programmer and am planning on using
> this cluster as a starting point to do research on building an ideal cluster
> for Animation for my own personal/entrepreneurial work. It would reside in
> my art studio. As an artist the idea of rack mounting the commodity PCS is
> much more fun that piling up the PCS.
> 
> I was thinking of working with a local hardware friend and figuring out how
> to screw on motherboards onto hardware type racks. Im sure there are better
> tried and tested racks out there that are not expensive. Any suggestions on
> the actual physical hardware for constructing racks for upto 16PCs.
> 
> Also any thoughts on racks versus piles of PCS.
> 
> A lot of the posts on the internet are old and out of date. I am wondering
> what the upto date trends are in racking commodity computers to create
> beowulf clusters. What should i be reading?

Look in the online archives -- they aren't old or out of date at all.
We just had a brief discussion of rack cases vs tower cases last week,
for example.  The consensus view from that was (that rack cases and
racks tend to be more expensive than tower cases and cheap shelving,
that 2U cases were likely to be quieter and less fussy than 1U cases,
that there some very "nifty" relatively new micro- form factor cases
that work quite well and attractively in shelved machine room
environments (suggesting that racks aren't the only way to get an
"artistically clean" looking cluster:-), and that either rack or micro
is likely to produce a smaller footprint cluster than the old/classic
shelf full of towers model.  Bladed systems were (as always) mentioned
as an alternative and (as always) it was pointed out that bladed systems
are an alternative for the truly deep pocketed as they are even more
expensive (if more compact) than racked systems.

There are enormous clusters built on all of these models.  It sounds
like you are interested in building a rendering farm.  The original
render farms for the original rendered cartoon movies were IIRC shelves
full of towers, as is IIRC Google, but I'm sure that a lot of them now
are racked up.

As for "trends" -- I doubt that there are any.  Beowulfery is all about
designing a cluster to meet your specific needs given your specific
application space and budget.  A rackmount cluster has certain
advantages, but they cost a certain amount extra.  At some point you
have to face the question of whether you are better off in the long run
spending the extra for rackmount boxes or would prefer to get cheaper
form factors and get more systems.  I hesitate to make pronouncements on
what SHOULD differentiate these choices as no matter what I say there
will be somebody on list who chose differently, quite probably for good
reasons.

So with a LARGE grain of salt, I'd say that very very loosely, if you
are building your first cluster, a hobby cluster, a low-budget cluster,
a small cluster (say less than 32 nodes total), or a production cluster
in an environment with lots of physical room and AC/power resources, one
or more shelf units of towers is either optimal or perfectly reasonable.

If you are a professional with experience building a commercial-grade
production cluster, especially one expected to have >=32 nodes in a real
machine-room environment, and you aren't horribly constrained in your
budget, you're more likely to go with rackmount FF nodes, or in the
richest and most space-constrained environments, even blades.

But these lines and differentiators are FAR from sharp.  I'm sure there
are people on list with 100's of shelved nodes (some of them just posted
in last week's discussion).  There are also bound to be people with
racks containing just four or five nodes (somewhat more likely if their
four or five nodes are just SOME of the systems in the preexisting
rack).

At home thus far I've tended to go with towers, at Duke I started with
towers years ago but now would only get rackmounts, and I've thought
pretty seriously about getting e.g. a half-height rolling rack for home
and starting to populate it a few U a year with my very limited budget.
The obstacle is that racks are expensive enough that it will cost me AT
LEAST one node just to get set up with a rack and a single rackmount
system, compared to just buying two equivalent power towers and popping
them into my existing $60 steel shelving.

OTOH, I could get at least four cores even in a single rackmount
chassis, for cheap.  OTOOH, I could probably get eight cores one way or
another in towers.  And while it isn't exactly "my money" I'm spending,
the particular pocket of OPM I'm using is quite finite.  So ultimately,
your decision here will come down to what you want to spend and what you
expect to get for it.  Beauty?  Ease of maintenance and access?  A
professional look to attract investors?  All of these things bear, which
is why the choice is not simple.

As far as books are concerned, I'll let others answer.  My online book
is free (so it costs you nothing to start there) but I'm the first to
admit that it is dated at this point, especially in its (lack of)
treatment of the more advanced networks.  Clustermonkey resources are
arguably more up to date, also free.  Many of the print books on the
subject are either similarly out of date or are written by people I've
never heard of, which basically means that they don't frequent this list
and participate in it, which means that I am skeptical about their
value (of the books).  The ones I've picked up in the store to thumb
through have mostly been pretty forgettable.

> Best regards,
> arjuna
> http://www.brahmaforces.com

Two names near and dear to my heart, given that I love the Mahabharata
and named my first cluster "Brahma" as well...;-)

    rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu