[Beowulf] Building my own highend cluster

[Brian Dobbins]

Hi Vincent (and others),

  I just wanted to add my own two cents after having fairly recently
recalled the relative complexity of creating diskless nodes 'by hand' a
few years back and subsequently finding the wonderful simplicity of
tools such as Warewulf (or Rocks).  So, in the interest of providing
more information to the discussion at hand, here's a bit more detail and
other assorted thoughts:

>[From pauln]
>.. my apologies in advance:
>http://www.psc.edu/~pauln/Diskless_Boot_Howto.html

  While I think cfengine and custom scripts gives a ton of flexibility, 
I've found it much easier on our diskless clusters to use the Warewulf
software ( http://www.warewulf-cluster.org/ ).  It handles a lot of the
behind-the-scenes dirty work for you (ie, making the RAM disks/tmpfs,
configuring PXE & DHCP, etc.) and the people on the mailing list tend to
be quick to respond to troubles with effective solutions.  Also, it's
actively supported by other people and it just makes life a lot easier,
in my opinion.  It isn't hard at all to tweak, either, and I'd happily
go into more detail if you wish, but I'd really recommend a quick look
through the website as well, just to get a rough idea of the process.

  Secondly, though I haven't used it myself, I recently spoke with a
friend who was very knowledgeable about Rocks, which also has a diskless
mode, I'm told.  Here's the link for that: (
http://www.rocksclusters.org/ )

> Programming in MPI/SHMEM is by the way pretty retarded way to program.

  If ease-of-use and shared-memory style are more important to you than
performance, you might be interested in checking out the "Cluster
OpenMP" developments in the Intel compilers.
http://cache-www.intel.com/cd/00/00/28/58/285865_285865.pdf

  The list even had a short discussion on it that if you missed starts here:
http://beowulf.org/pipermail/beowulf/2006-May/thread.html#15543

> Shared memory programming is just *so superior* over that in terms of ease
> for the programmer and it also means you can just recompile your code to
> work on a quad opteron or 8 processor opteron without any need to strip
> MPI commands.

  This is mostly an aside, but why would you need to strip MPI commands to
run on a 4 or 8 processor system?  Or a one processor system, for that
matter.  I agree shared memory methods are easier to program, but I
almost never need to strip MPI commands from my programs.  In the worst
case, I may need to add/remove a flag at compile time and use #defines
in the code, but whenever possible, I actually try to build a code with
MPI libraries that has a switch ('-serial') to allow it to run in
single-processor mode.  It just calls different routines where
communication would ordinarily be.  It's nice, elegant, and probably
makes some people gasp in horror because of a few checks the code needs
to make which possibly impact performance in a negligible way, but it's
certainly extremely handy when benchmarking and verifying results.

  Finally, going back to the beginning of the discussion, I'd just caution
you about putting motherboards on a slab of wood in a garage.  The
filter might keep dust out of the garage, but other things always seem
to manage to get into garages, and lots of creepy-crawly things love
warmth and light - two things your system are bound to give off.  :)

  Good luck and have fun,
  - Brian