[Beowulf] Network Filesystems performance

[Glen Dosey]

I am having issues with read performance on our cluster using NFS ( and
also SMB) . Any initial file transfer to the clients seems to be limited
to ~40MB/s. This is not a limitation of the underlying array, as I can
read the same file on the server at ~160MB/s. Similarly, after the file
has been read into memory on the server, the clients can read it at
~100MB/s (this is following a NFS umount and mount on the client). The
clients can write to the server at ~90MB/s. 

All of this leads me to believe the issue I am experiencing is not
directly network or disk related, but rather a factor of the network
latency combined with disk request latency, such that the client is not
sending enough read requests (fast enough) to keep the server disk
busy. 

I looked into a method to have the client or server NFS perform a read
ahead such that it would keep the disk busy and alleviate the latency
performance hit I suggest above. Perhaps someone can offer a solution or
point me in a different direction as the above may not really be the
issue. 

I have tried adjusting a number of kernel parameters to increase the TCP
window size but it has not had any affect.

Thanks,
Glen


I've left out a lot of detail to keep this succinct. I can provide it
when necessary.