Advice for 2nd cluster installation

[Robert G. Brown]

On Fri, 10 Jan 2003, Mike Eggleston wrote:

> I'm part of a startup group that will be using a larger cluster soon. I
> currently have a very ad-hoc cluster made of old machines I could
> scrounge for the task. What I want to do, mostly because of the money
> involved, is buying motherboards from newegg.com, adding cpu, fan, power
> supply, and memory, putting the whole thing in some sort of rack or
> enclosure to keep it neat, and hook it to a switch. Given this low-tech
> idea does anyone have any ideas or suggestions on how to rack/house
> these boards? Once installed I will also redirect an A/C plenum(?)/duct
> to directly above the rack. Space is not so much at a premium as
> simple capital at this point.

For a 16 node cluster that is as cheap as humanly possible, see pictures
on the brahma site.  The cluster in the foreground of the left picture
is:

  1) sixteen mid-tower single CPU 1200 MHz athlons, configured at about
$900 each almost two years ago.  Nowadays a 1600 GHz athlon or 2.4 GHz
P4 would cost about what, $700 in an identical configuration OTC.
Hand-built you could probably get it down to $600, maybe spending a bit
of CPU speed.  A variety of possible motherboards you could use would
have onboard 100BT NIC and video so that wouldn't be a problem.
  2) $50 heavy duty steel shelving from Home Depot.  I wasn't kidding.
As it is it we set the shelving up in two half-units, but we've rebuilt
this so they all sit in the single unit with half the footprint.
  3) 1 16 port Netgear switch.  I think it currently costs around $100
or a bit less.
  4) a few surge protectors, some cable ties, some ethernet cables, and
of course somewhere you need a "server/head" node and a KVM.

If you buy them prebuilt and we estimate $750 each with an extra $250
for KVM and more disk on the head node, that is 15x750 + 1000 = $12,250
for the systems.  $50 for the shelving, maybe $200 (being VERY generous)
for everything else, total of $12,500.

At $600/node with a local disk you can drop this to $10,250 (for 16
nodes including the head node).

At $500/node (which will probably mean diskless nodes, even hand built,
unless you get really cheap e.g. celeron or duron motherboards and
minimal memory) you could get it nicely under $9000 (maybe $550/node
including everything), but you might start giving up speed per node.

The problem with any other configuration e.g. rackmounts is that
rackmount cases tend to cost roughly $200 more than cheap but adequate
tower cases, and a full height two-post rack (no "enclosure", bare
bones) costs about $200 (including screws and maybe a shelf) and has to
be securely bolted into the floor to be safe when loaded, which is a lot
of work involving power tools and so forth for most floors.  So you can
count on it costing you at LEAST $3200 more in any equivalent
configuration.

Blade configurations tend to add on even to that, especially when you
take into account their often significantly slower CPUs.  A cheap 2.4 P4
can be pretty fast...compared to an 800 MHz P3.

   rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu