[Beowulf] Should I go for diskless or not?
larry.stewart at sicortex.com
Fri May 15 03:43:08 PDT 2009
>> On 5/13/09, Dr Cool Santa <drcoolsanta at gmail.com> wrote:
>>> I have a cluster of identical computers. We are planning to add more
>>> later. I was thinking whether I should go the diskless nodes way or not?
>>> Diskless nodes seems as a really exciting, interesting and good option,
>>> however when I did it I needed to troubleshoot a lot. I did fix it up, but
>>> had to redo the filesystem, but the past experiences didn't make much of
>>> difference. I still need to fix up everything, I kinda need your help to
>>> Also, performance wise, I was thinking that diskless is not a good
>>> and since performance matters . . .
>>> Can somebody outline the pros and cons of each or just give me thier
We run diskless on our systems, up to 972 nodes. Necessarily, because
we don't have local disks.
On small systems we run NFS root and on larger ones we run about 200
nodes per root-server, using nbd.
I've measured the root working set up to being ready to accept jobs, and
it is around 140 MB worth of root traffic, which isn't too bad.
I'll echo the remarks about swapping, there is a large patch set for
swapping over IP, and we don't run that. In fact right now we run
without swap space, and vm_overcommit_ratio set to "90". This is
generous enough that we're not having problems, even on large systems,
with running out of memory. Everyone seems to agree that having some
swap space is good for stability, so we do plan to add swap at some
point. We've got a new network block device that can swap over the
interconnect (without any allocations) at about 2 GB/s which is probably
good enough to make DSM interesting. If you have local disks, using
them for swap will work fine.
Because using nbd forces a read-only rootfs, we have a somewhat modified
root filesystem, with extra symlinks for the stuff that can't be
read-only but even this slight amount of non-standardness makes
installers for some packages unhappy, some are not prepared for symlinks
in the paths to their components. We're evaluating using unionfs on top
of the read-only root to make that easier. The alternative is making
the new nbd server do copy-on-write.
More information about the Beowulf