From owner-freebsd-fs@FreeBSD.ORG Tue Jul 19 23:09:41 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D490106566B for ; Tue, 19 Jul 2011 23:09:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id BD2C88FC17 for ; Tue, 19 Jul 2011 23:09:40 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAHINJk6DaFvO/2dsb2JhbABUG4QvpAi7OpEtgSuEAoEPBJJoiC2ISQ X-IronPort-AV: E=Sophos;i="4.67,231,1309752000"; d="scan'208";a="131626738" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 19 Jul 2011 19:09:39 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A6BB4B3F1F; Tue, 19 Jul 2011 19:09:39 -0400 (EDT) Date: Tue, 19 Jul 2011 19:09:39 -0400 (EDT) From: Rick Macklem To: Mike Shultz Message-ID: <732996961.772204.1311116979640.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4E24B266.9050108@votesmart.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org, Clinton Adams Subject: Re: nfsd server cache flooded, try to increase nfsrc_floodlevel X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jul 2011 23:09:41 -0000 It's me again: > I ran into an issue today of our server thinking that it was being > flooded and locking our nfs users out. Got a LOT of these messages: > > Jul 12 16:08:22 xxxxx kernel: nfsd server cache flooded, try to > increase > nfsrc_floodlevel > > Our server(`uname -a`): FreeBSD xxxxx 8.2-RELEASE-p2 FreeBSD > 8.2-RELEASE-p2 #0: Tue Jun 21 16:52:27 MDT 2011 > yyy@xxxxx:/usr/obj/usr/src/sys/XXXXX amd64 > > I could find no information on nfsrc_floodlevel other than source code > which didn't explain too much about it. I don't know if it's a kernel > config var, or what. > > `nfsstat -e` did show this: > > CacheSize TCPPeak > 16385 16385 > > So I'm guessing that that is the current cache limit. > > The source code and this output suggest that we're just running into > the > limit. However, a comment in that source does suggest that "The cache > will still function over flood level" but that doesn't seem to be the > case. I ended up having to revoke the clients and restarting nfsd to > get > it operational again. > I've created a patch that gets rid of open_owners (and their associated cached replies) agressively when the count hits about 90% of nfsrc_floodlevel. A quick test here indicates it allows the server to "recover" without hitting the flood gate. Note that this is allowed by the NFSv4.0 RFC, so the change doesn't break the protocol. Please try the patch, which is at: http://people.freebsd.org/~rmacklem/noopen.patch (This patch is against the file in -current, so patch may not like it, but it should be easy to do by hand, if patch fails.) Again, good luck with it and please let me know how it goes, rick