diskless nodes? (was Re: Xbox clusters?)

[Velocet]

On Thu, Dec 06, 2001 at 09:58:53AM +0100, David van der Spoel's all...
> On Wed, 5 Dec 2001, Velocet wrote:
> 
> >Wondering why everyone gets local drives here - do you people have
> >computational software that needs to write scratch or swap faster than
> >100Mbps (12.5MB/s) or even GBE (125MB/s theoretical, 35-50MB/s actual)? Isnt
> >that fast enough?
> >
> >Why arent diskless netbooting clusters more popular?
> Because you should divide the real disk througput on the server by the
> amount of nodes, plus add the latency for setting up a connection.

NFSv3 isnt all that slow for setting up a connection, I dont think.  If your
argument is opening and closing lots of little files extremely quickly, then
you'd be relying on your caching in your OS to speed things up. (I'm not sure
if linux has something like freeBSD's softupdates setting for filesystem
mounting.)

> If you have only slightly disk-intensive applications (e.g. quantum
> chemistry) this will severely slow down your calculations. We are using a
> home-grown installation, basically redhat kickstart. We take 1 Gb for the
> OS, 2 Gb swap and the rest (6-30 Gb) /tmp. That works nicely for us and
> disks are cheap enough.

Do you have alot of swapping to do locally? If you are using cheap disks, are
you getting much > 12.5MB/s throughput? At this point it wouldnt matter all
that much for management as if the local disk is just 30 gigs of swap space or
scratch space, you can slam a local drive in and still mount the OS over NFSv3
to save management headaches for updating, adding new configs and software,
etc. I'd assume you dont need to load different chunks of the OS very often,
and even so you'd be relying on caching again.

I assume you're talking about gromacs. Does gromacs really need 12.5MB/s
constantly throughout the calculation, or just in spurts? (In our usage of it
we saw very little disk access other than logging mainly.) And how much disk
footprint does it ultimately need for what you're doing (I suppose a job that
requires infinite disk can probably be constructed, but so far in my limited
use of it, I've never required anything like 100 or even 10 gigs to
keep jobs running).

As I said before, for a low-intensity disk access operation (on the order
of an AVERAGE (not 'desired' burst rate) of 1-5MB/s) then diskless clusters
for OS, swap and /tmp make alot of sense. Our usage of G98 and gromacs
fit into this category (for now ;)). I am sure a number of others' use
of clusters also fit into this range, thus my wondering.

I was thinking it was the lack of knowledge more than anything... diskless
swap can be tricky (though its really not in freebsd :)

/kc


> 
> Groeten, David.
> ________________________________________________________________________
> Dr. David van der Spoel, 	Biomedical center, Dept. of Biochemistry
> Husargatan 3, Box 576,  	75123 Uppsala, Sweden
> phone:	46 18 471 4205		fax: 46 18 511 755
> spoel at xray.bmc.uu.se	spoel at gromacs.org   http://zorn.bmc.uu.se/~spoel
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 

-- 
Ken Chase, math at velocet.ca  *  Velocet Communications Inc.  *  Toronto, CANADA