diskless nodes? (was Re: Xbox clusters?)

[Troy Baer]

On Fri, 7 Dec 2001, Velocet wrote:
> On Fri, Dec 07, 2001 at 09:47:10AM -0500, Troy Baer's all...
> > Well, there's nothing keeping you from keeping the root filesystem on NFS
> > and using local disk for swap, /tmp, and /var.  We do that on our Pentium
> > III and Itanium clusters, and it seems to work pretty well.  The biggest
> > problem is user education ("No, 'cp file /tmp' does not copy it out to all
> > the nodes in your job, use pbsdcp instead").
> 
> With large scratch disk bandwidth required I'd definitely go this way.
> But even with $100-$150 a node on $300 diskless nodes, thats 35-50% the
> cost of a node.

That's true, *if* you're buying $300 nodes.  I'm not, though; our node
cost tends to be around $2500-3000, because we tend to buy server-class
SMP mobo's, lots of memory, Myrinet, rackmount cases, and a bunch of other
stuff to keep me from having to walk/drive over to the machine room (in a
secured building about 1.5 miles away) every time I need to reboot nodeXX.
In this situation, the cost of the disk (~$2.50/GB these days for IDE) is
down in the noise, and it's a big performance boost for disk hogs like
Gaussian.

> If the parameters of the requirements for the
> cluster's disk usage dont dictate more than a few Mbps of scratch
> bandwidth being required, then you can get a much higher price
> performance out of diskless machines. Of course this reduces
                                                  ^^^^^^^^^^^^
> flexibility of the cluster, but if you can ensure its only used for
  ^^^^^^^^^^^^^^^^^^^^^^^^^^
> certain types of jobs that fit into a range of parameters, then you
> can get a fair increase in performance for the money.

Reducing flexibility is not an option in the environment I work in,
because our users cover an enormous range of applications.  Performance/price
ratio is not our main design goal, merely a happy side effect. :)

	--Troy
-- 
Troy Baer                       email:  troy at osc.edu
Science & Technology Support    phone:  614-292-9701
Ohio Supercomputer Center       web:  http://oscinfo.osc.edu