Date: Fri, 15 May 2009 07:51:33 +0200 (CEST) From: Konrad Heuer <kheuer2@gwdg.de> To: Robert Watson <rwatson@FreeBSD.org> Cc: freebsd-hackers@freebsd.org Subject: Re: How to invalidate NFS read cache? Message-ID: <20090515073949.M70549@gwdu60.gwdg.de> In-Reply-To: <alpine.BSF.2.00.0905121948180.71532@fledge.watson.org> References: <20090508101555.J47014@gwdu60.gwdg.de> <alpine.BSF.2.00.0905121948180.71532@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 12 May 2009, Robert Watson wrote: > On Fri, 8 May 2009, Konrad Heuer wrote: > >> sporadically, I observe a strange but serious problem in our large NFS >> environment. NFS servers are Linux and OS X with StorNext/Xsan cluster >> filesystems, NFS clients Linux and FreeBSD. >> >> NFS client A changes a file, but nfs client B (running on FreeBSD) does >> still see the old version. On the NFS server itself, everything looks fine. >> >> Afaik the FreeBSD kernel invalidates the NFS read cache if file >> modification time on the server changed which should happen here but >> doesn't. Can I force FreeBSD (e.g. by sysctl setting) to read file buffers >> again unconditionally after vfs.nfs.access_cache_timeout seconds have >> passed? > > Hi Konrad: > > Normally, NFS clients implement open-to-close consistency, which dictates > that when a close() occurs on client A, all pending writes on the file should > be issued to the server before close() returns, so that a signal to client B > to open() the file can validate its cache before open() returns. > > This raises the following question: is client A closing the file, and is > client B then opening it? > > If not: relying on writes being visible on the client B before the close() on > A and a fresh open() on B is not guaranteed to work, although we can discuss > ways to improve behavior with respect to expectation. Try modifying your > application and see if it gets the desired behavior, and then we can discuss > ways to improve what you're seeing. > > If you are: this is probably a bug in our caching and or issuing of NFS RPCs. > We cache both attribute and access data -- perhaps there is an open() path > where we issue neither RPC? In the case of open, we likely should test for a > valid access cache entry, and if there is one, issue an attribute read, and > otherwise just issue an access check which will piggyback fresh attribute > data on the reply. Perhaps there is a bug here somewhere. > > A few other misc questions: > > - Could you confirm you're using NFSv3 on all clients. Are there any special > mount options in use? > - What version of FreeBSD are you running with? > > In FreeBSD 8.x, we now have DTrace probes for all of the above events -- > VOPs, attribute cache hit/miss/load/flush, access cache hit/miss/load/flush, > RPCs, etc, which we can use to debug the problem. I haven't yet MFC'd these > to 7.x, but if you're able to run a very fresh 7-STABLE, I can probably > produce a patch to add it for you in a few days. Hello, Robert, thank you very much for your reply! The problem I observe happens with FreeBSD 6.4-R and 7.0-R with nfsv3. The fstab entry I use is: server:/Volume /local/dir nfs bg,rw,intr,-T,-r32768,-w16384 0 0 The server runs on Mac OSX 10.5. In the meantime, I had the chance to examine a failure a little bit closer. As far as I can see in the moment a file modified on a Linux NFS client gets a new modification time on the NFS server but the FreeBSD client still sees the old timestamp. This obviously happens sporadically only under some circumstances I do not know further. I'll do some further testing the next days. Could you imagine a kind of directory or metadata caching on FreeBSD NFS clients that may cause this behaviour? Best regards Konrad Konrad Heuer GWDG, Am Fassberg, 37077 Goettingen, Germany, kheuer2@gwdg.de
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090515073949.M70549>