[Beowulf] Troubleshooting NFS stale file handles

[Ryan Novosielski]

Are you saying they can’t mount the filesystem, or they can’t write to a mounted filesystem? Where does this system get its user information from, if the latter?

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Apr 19, 2017, at 12:09, Prentice Bisbal <pbisbal at pppl.gov> wrote:
> 
> Beowulfers,
> 
> I've been trying to troubleshoot a problem for the past two weeks with no luck. We have a cluster here that runs only one application (although the details of that application change significantly from run-to-run.). Each node in the cluster has an NFS export, /local, that can be automounted by every other node in the cluster as /l/hostname.
> 
> Starting about two weeks ago, when jobs would try to access /l/hostname, they would get permission denied messages. I tried analyzing this problem by turning on all NFS/RPC logging with rpcdebug and also using tcpdump while trying to manually mount one of the remote systems. Both approaches indicated state file handles were prevent the share from being mounted.
> 
> Since it has been 6-8 weeks since there were any seemingly relevant system config changes, I suspect it's an application problem (naturally). On the other hand, the application developers/users insist that they haven't made any changes, to their code, either. To be honest, there's no significant evidence indicating either is at fault. Any suggestions on how to debug this and definitively find the root cause of these stale file handles?
> 
> -- 
> Prentice
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf