[Beowulf] Should I go for diskless or not?

[Lawrence Stewart]

>>
>> On 5/13/09, Dr Cool Santa <drcoolsanta at gmail.com> wrote:
>>     
>>> I have a cluster of identical computers. We are planning to add more
>>> nodes
>>> later. I was thinking whether I should go the diskless nodes way or not?
>>> Diskless nodes seems as a really exciting, interesting and good option,
>>> however when I did it I needed to troubleshoot a lot. I did fix it up, but
>>> I
>>> had to redo the filesystem, but the past experiences didn't make much of
>>> a
>>> difference. I still need to fix up everything, I kinda need your help to
>>> decide.
>>> Also, performance wise, I was thinking that diskless is not a good
>>> option,
>>> and since performance matters . . .
>>> Can somebody outline the pros and cons of each or just give me thier
>>> opinion.
>>>
>>>       
We run diskless on our systems, up to 972 nodes.  Necessarily, because 
we don't have local disks.
On small systems we run NFS root and on larger ones we run about 200 
nodes per root-server, using nbd.
I've measured the root working set up to being ready to accept jobs, and 
it is around 140 MB worth of root traffic, which isn't too bad.

I'll echo the remarks about swapping, there is a large patch set for 
swapping over IP, and we don't run that.  In fact right now we run 
without swap space, and vm_overcommit_ratio set to "90".  This is 
generous enough that we're not having problems, even on large systems, 
with running out of memory.  Everyone seems to agree that having some 
swap space is good for stability, so we do plan to add swap at some 
point.  We've got a new network block device that can swap over the 
interconnect (without any allocations) at about 2 GB/s which is probably 
good enough to make DSM interesting.  If you have local disks, using 
them for swap will work fine.

Because using nbd forces a read-only rootfs, we have a somewhat modified 
root filesystem, with extra symlinks for the stuff that can't be 
read-only but even this slight amount of non-standardness makes 
installers for some packages unhappy, some are not prepared for symlinks 
in the paths to their components.  We're evaluating using unionfs on top 
of the read-only root to make that easier.  The alternative is making 
the new nbd server do copy-on-write.

-Larry/SiCortex