[Beowulf] NFSv3 client hangs - tcp v/s udp.
jlb17 at duke.edu
Thu May 11 07:35:57 PDT 2006
On Wed, 3 May 2006 at 5:21pm, Amitoj G. Singh wrote
> After upgrading from Red Hat 7.1 to Red Hat EL 4 we realized that we were
> having a 1 in 10 user jobs fail because of a worker node NFS mount point
> failing to respond. The NFS mount points on the worker nodes would become
> unresponsive during heavy NFS I/O. A simple "netstat -t" on the
> head-node showed that there were thousands of open TCP nfs sockets on the
> head-node. Worker nodes that had frozen NFS mount points responded with
> the following error message:
> nfs_statfs: error no = 512
I had a discussion with Trond about that error on the NFS list back in
December. That error essentially means that somebody (or something)
interrupted the RPC call. E.g., you type 'df', it hangs waiting for a NFS
mount, and you 'CTRL-C' it -- you'll see that error. The error message
was planned to be removed in 2.6.16. IOW, it's a symptom, not the
> We recently switched all our NFS mounts to use udp and have had no worker
> nodes with failing or unresponsoive NFS mount points.
> Thought would share this bit of experience with the list. Interestingly
> while googling we did not find a lot of chatter about this issue.
I've seen some discussions of this on nahant-list as well as the NFS list.
The problem is that it's hard to easily reproduce. If you have a test
case and a support contract, I'd heartily recommend getting in touch with
RH about it directly.
Department of Biomedical Engineering
More information about the Beowulf