From owner-freebsd-stable@FreeBSD.ORG Sat Dec 4 23:39:18 2010 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8A371065673 for ; Sat, 4 Dec 2010 23:39:18 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mail.egr.msu.edu (boomhauer.egr.msu.edu [35.9.37.167]) by mx1.freebsd.org (Postfix) with ESMTP id 98F798FC1E for ; Sat, 4 Dec 2010 23:39:18 +0000 (UTC) Received: from boomhauer (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id A695B7D39E; Sat, 4 Dec 2010 18:39:17 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by boomhauer (boomhauer.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ItXGKk1M+AaB; Sat, 4 Dec 2010 18:39:17 -0500 (EST) Received: from daemon.localdomain (daemon.egr.msu.edu [35.9.44.65]) by mail.egr.msu.edu (Postfix) with ESMTP id 738837D397; Sat, 4 Dec 2010 18:39:17 -0500 (EST) Received: by daemon.localdomain (Postfix, from userid 21281) id 6BA9E1309C5; Sat, 4 Dec 2010 18:39:17 -0500 (EST) Date: Sat, 4 Dec 2010 18:39:17 -0500 From: Adam McDougall To: Rick Macklem Message-ID: <20101204233906.GY82436@egr.msu.edu> References: <4CF5DEC4.3070901@egr.msu.edu> <1918825226.1032577.1291246386604.JavaMail.root@erie.cs.uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1918825226.1032577.1291246386604.JavaMail.root@erie.cs.uoguelph.ca> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: stable@freebsd.org Subject: Re: Stale NFS file handles on 8.x amd64 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Dec 2010 23:39:18 -0000 On Wed, Dec 01, 2010 at 06:33:06PM -0500, Rick Macklem wrote: > > I'll give negnametimeo=0 a try on one server starting tonight, I'll be > busy tomorrow and don't want to risk making anything potentially worse > than it is yet. I can't figure out how to disable the attr cache in > FreeBSD. Neither suggestions seem to be valid, and years ago when I > looked into it I got the impression that you can't, but I'd love to be > proven wrong. I just looked and, yea, you are correct, in that the cached attributes are still used while NMODIFIED is set if the mtime isn't within the current second. (I'm not going to veture a guess as to why this is done at this time:-) But, "acregmon=0,acregmax=0,acdirmin=0,acdirmax=0" looks like it comes close, from a quick inspection of the code. I haven't tested this. You do have to set both min and max == 0, or max just gets set to min instead of 0. The *dir* ones apply to directories and the *reg* ones otherwise. Ok. I'm applying it right now (with s/mon/min/) > I'll try dotlock when I can. Would disabling statd and > lockd be the same as using nolock on all mounts? Nope. If you kill off lockd and statd without using the "nolock" option, I think all file lock operations will fail with ENOTSUPPORTED whereas when you mount with "nolock", the lock ops will be done locally in the client (ie seen by other processes in the same client, but not by other clients). This last go around I told dovecot to use dotlock for more things, not just Maildir files (apparently) and it apparently made it stop using fcntl because it no longer complained when I temporarily stop lockd. It did not help the Stale files through, nor did turning off lockd, so I'm still hunting. > The vacation binary > is > the only thing I can think of that might use it, not sure how well it > would like missing it which is how I discovered I needed it in the > first > place. Also, if disabling lockd shows an improvement, could it lead to > further investigation or is it just a workaround? Well, it's a work around in the sense that you are avoiding the NLM and NSM protocols. These are fundamentally flawed protocol designs imho, but some folks find that they work ok for them. Imho, the two big flaws are: 1 - Allowing a blocking lock in the server. Then what happens if the client is network partitioned when the server finally acquires the lock for the client? (NFSv4 only allows the server to block for a very short time before it replies. The client must "poll" until the lock is available, if the client app. allows blocking. In other works, the client does the blocking.) 2 - It depends upon the NSM to decide if a node is up/down. I'm not sure what the NSM actually does, but it's along the lines of an IP broadcast to see if the other host(s) are responding and then sets up/down based on how recently it saw a message from a given host. (NFSv4 requires that the server recognize a lock request where the client had state that predates this boot and reply with an error that tells the client to recover its lock state using special variants of the lock ops. Imho, this does a much better job of making sure the server and clients maintain a consistent set of lock state. The server may throw away lock state for an NFSv4 client if it hasn't renewed the state within a lease time and then the client will be given an "expired" error to tell it that it has lost locks. This should only happen when a network partitioning exceeds the lease duration and, in the case of the FreeBSD NFSv4 server, it has also received a conflicting lock request from another client whose lease has not expired.) I always appreciate learning more. I have a fairly strict firewall in place and don't see any denies related to rpc/nfs but I'll keep my eyes open. I'm presently allowing all traffic between the client and the nfs server on all ports, tcp and udp. The firewall was put in place after the problems started, as part of a different project. Probably a lot more glop than you expected, but I couldn't resist a chance to put in a plug for NFSv4 file locking. Btw, you could try NFSv4 mounts, since Netapp and the experimental FreeBSD8 client both support them. I'll keep that in mind too, I tried it a while back before the revamp. >From a firewall standpoint (all around), I would love to get rid of v3 when the time comes. Although, I would probably invest my time into testing dovecot v2 where the author tries to keep users on their own server anyway. Either would probably be a bigger project. I have some solaris clients that are too smart for their own good for example and broke when I enabled v4 on the netapp in the past (permission issues). Good lock (oh, I meant luck:-) with it, rick HA ha :)