[Beowulf] diskless booting over jumbo frames
Bogdan.Costescu at iwr.uni-heidelberg.de
Wed Apr 25 09:30:14 PDT 2007
On Wed, 25 Apr 2007, Amrik Singh wrote:
> I agree that Jumbo Frames would not be a great help with the root
> file system but we hope to get a better performance from other NFS
I don't quite follow you here: the link that was sent earlier had
shown some kernel level autoconfig problem. If this is indeed still
the case with newer kernels, this should only affect the root FS -
before mounting the other NFS exports, you should have the chance to
perform a proper initialization of the network, which would give you
the possibility to use a larger MTU.
> As all the machines on the same subnet have to be using the jumbo
Why ? The MTU specifies a _maximum_ value; even when using 1500, not
all packets are exactly 1500 bytes. The larger MTU _allows_ larger
values, but doesn't _force_ them.
> Even though NFS is extremely slow, copying files over scp is still
> very fast between a client and server.
If by this you mean that NFS is slower than scp, then this should be
exactly the other way around, because scp needs CPU time for its
encryption. I would have said that you have some network problems and
use NFS over UDP which leads to high retransmission rates; scp would
adapt better to the network problems due to using TCP... but you just
mention that you tried switching between TCP and UDP.
> We have tried all different ways to tune the NFS for a better
> performance (increasing NFS deamons on the servers, changing rsize &
> wsize, using TCP vs UDP, using async vs sync, noatime, timeo).
Could it be that you tried so hard to tune it that you just used too
many settings that don't play well together in your particular
situation ? I've found the NFS client and server in the kernels of the
recent years to perform reasonably well in their default
configuration; although they could be further optimized, a NFS
transfer in these conditions would always beat a scp one.
> initrd=bzImage ramdisk=40960
dhclient could be located in the ramdisk. Actually even a whole root
FS could be located in the ramdisk, if there are not too many nodes
booting at the same time and leading to UDP packet loss.
... and you already have a rather large ramdisk. Have you created it
yourself (and therefore know what's in it and how to add some more) ?
IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De
More information about the Beowulf