[Beowulf] best archetecture / tradeoffs

[Mark Hahn]

> This VNFS resides on the master/boot server, and is used to construct
> the root filesystem for each node in your cluster.  You can make changes
> to this template directly using chroot, or indirectly with other scripts/tools
> for example:  rpm --root /vnfs/default ..., or yum
> --installroot=/vnfs/default ...

heh, that's exactly what I do, but never thought to give it a name.
or rather, I thought "nfsroot" pretty well covered it (and rejoiced
that rpm/yum have those switches.)

so in what sense is it a virtual node FS?  how is it different from 
the fairly common practice of an NFS-mounted root filesystem?

> recent versions, we introduced a hybrid NFS/ramdisk scheme that reduces
> the permanent RAM footprint dramatically by using a readonly NFS mount
> of the VNFS for non-critical files. Thus, you can have your full blown text

presumably just most bits of /var, no?  are there less obvious bits that 
you feel need to be in the ramdisk?  incidentally, do you use a ramdisk,
or do you use initrd's cpio format, or do you simply populate a tmpfs 
during the boot process?  I do the latter - it seemed simplest, once the 
initrd is under way, and has the NFS root mounted, to just mount tmpfs 
here and there and untar (the tar is in the NFS too...)

> editor, compiler, and X installed in the VNFS, yet still have only a 15 to
> 30 MB ramdisk on the nodes.  Which files reside on the ramdisk vs. which

hmm, 15M seems fairly elaborate, or do you not pivot/umount away your 
boot code?

> created.  Thus, you can get much of the small-RAM-footprint benefit of
> the NFS-root scheme, yet have dramatically lower NFS traffic to the server
> during normal cluster use.

I'd heard people say that was a problem, but haven't found it so.  what files
are inadequately cached by NFS and wind up causing noticable traffic?
it seems like starting a new job would read little more than the user's 
shell, some shared libraries, /etc/passwd and friends.  I haven't tried to 
collect traces, but they seem quite NFS-caching-friendly...

regards, mark hahn.