Date: Thu, 9 Oct 1997 19:40:11 +0000 (GMT) From: Terry Lambert <tlambert@primenet.com> To: karl@Mcs.Net (Karl Denninger) Cc: phk@critter.freebsd.dk, karl@Mcs.Net, robin@intercore.com, freebsd-current@FreeBSD.ORG Subject: Re: NFS cache problem in 3.0 SNAP Message-ID: <199710091940.MAA08049@usr08.primenet.com> In-Reply-To: <19971009132146.18909@Mars.Mcs.Net> from "Karl Denninger" at Oct 9, 97 01:21:46 pm
next in thread | previous in thread | raw e-mail | index | archive | help
> > I'm wondering if we need a call from the nfs code to the namecache > > when the nfs code decides that a directory has been changed. Anybody > > know where we can detect that situation ? > > Not offhand, no. > > The problem we see here which is very repeatable is this: > > Boxes (1 and 2, server is on system 3 which is not involved) > > 1: mv file1 file2 > 2: cat file1 (still works) The problem here is that the name cache is done at the VOP_LOOKUP level instead of being done at the nami() level. The name cache needs to be done at a higher level. The sole exception is n:m mapping layers, like unionfs, which must make the name cache calls for the underlying code. > 1: cp file3 file1 > 2: cat file1 (still shows OLD contents of file 1) Naturally; the cache entry is the vnode for the nfsnode. > 1: rm file1 > 2: cat file1 (STILL shows old contents - you've now lost the ability > to get at the handle which was there first!) > > This is bad. This one is really an issue with the name cache code itself. One way to solve this would be to treat the existance of a vnode in cache as if it were an open reference. The real problem here (with NFS) is "multiclient deletion idempotence"; specifically, if a file is open, and I delete it, this is what is supposed to happen: File open locally? Rename file and queue deletion File open by other client? I don't know; I assume it isn't: actually delete file Typical implementation will assert a lock on the file for each open instance so that the file will have a "reference count" some place stateful, so that multiple client interactions tend to result in less stale handles. > Here's another: > 1: rm file1 > 2: cat file1 (returns "Stale NFS handle" - not file not found!) The removal did not modify the local directory name cache, obviously. You must understand that a lookup followed by a FHTOVP will not see cache references as file references, and therefore you will get a different vnode. When the vnode is "purged from the cache", then it will be a vnode not in the cache, and the other parent/name pair to get the vnode will get a different vnode. This only happens if there are no open references to the vnode at the time. Another potential workaround for this would be to not use a global vnode pool, but instead make the vnode an element of the incore inode. In the NFS case, this is the incore nfsnode. If you did this, a forward reference for the vnode would always exist. > 1: cp file3 file1 > 2: cat file1 (now shows proper content; it "fixed itself") It did *not* "fix itself". The "stale file handle" response triggered throwing away the reference; when you did the copy, you instanced another reference. The second "cat" is using the second reference. I'm not positive, but I believe that this will be a replacement entry for an existing entry for which the VPTOFH results in "stale handle"; this may actually "leak" a vnode. I would actually prefer doing all three fixes, but in a pinch, making the directory cache entries counting references (just like opens) would do to mask the problem. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199710091940.MAA08049>