[Beowulf] Compute Node OS on Local Disk vs. Ram Disk

[Jon Forrest]

There are two philosophies on where a compute node's
OS and basic utilities should be located:

1) On a local harddrive

2) On a RAM disk

I'd like to start a discussion on the positives
and negatives of each approach. I'll throw out
a few.

Both approaches require that a compute node "distribution"
be maintained on the frontend machine. In both cases
it's important to remember to make any changes to this
distribution rather than just using "pdsh" or "tentakel"
to dynamically modify a compute node. This is so that the
next time the compute node boots, it gets the uptodate
distribution. Although the mechanism for maintaining
the distribution varies in either approach, I consider
this a push since one mechanism isn't inherently better
than the other.

Assuming the actual OS image is the same in both cases,
#2 clearly requires more memory than #1. There are actually
two approaches to #2 - a) where only the OS and other stuff
necessary to boot the system are kept in memory and everything
else is in an NFS-mounted file system, and b) where the whole OS
installation is kept in memory. Depending on which approach
is taken, the RAM-based installations can take hundreds of MB
more than a local harddrive installation. However, on a modern
multicore compute node this might just be a few percent of the
total RAM on the node.

Long ago not installing a local harddrive saved a considerable
about of money but this isn't true anymore. Systems that need
to page (or swap) will require a harddrive anyway since paging
over the network isn't fast enough so very few compute nodes
will be running diskless.

Approach #2 requires much less time when a node is installed,
and a little less time when a node is booted.

What are some of your favorite issues, positive or negative, with
each approach?

Cordially,
-- 
Jon Forrest
Research Computing Support
College of Chemistry
173 Tan Hall
University of California Berkeley
Berkeley, CA
94720-1460
510-643-1032
jlforrest at berkeley.edu