From owner-freebsd-stable@FreeBSD.ORG Fri Jul 25 00:15:10 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CEA7ED04 for ; Fri, 25 Jul 2014 00:15:10 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 946262984 for ; Fri, 25 Jul 2014 00:15:10 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArQEAHKg0VODaFve/2dsb2JhbABRCBaDSlcEgnTGPwqHRQGBJneEAwEBAQMBAQIgVhsOCgICDRkCKi8GiE0IDacTgROXNheBLI1IASI0B4J4gU4Fji2KGoRBih2IVoNkIS8BgQNB X-IronPort-AV: E=Sophos;i="5.01,727,1400040000"; d="scan'208";a="143515400" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 24 Jul 2014 20:14:02 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id BD625B408D; Thu, 24 Jul 2014 20:14:02 -0400 (EDT) Date: Thu, 24 Jul 2014 20:14:02 -0400 (EDT) From: Rick Macklem To: Harald Schmalzbauer Message-ID: <1327388853.3033655.1406247242764.JavaMail.root@uoguelph.ca> In-Reply-To: <53D0CBD6.1020708@omnilan.de> Subject: Re: nfsd server cache flooded, try to increase nfsrc_floodlevel MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 00:15:11 -0000 Harald Schmalzbauer wrote: > Bez=C3=BCglich Rick Macklem's Nachricht vom 08.08.2013 14:20 (localtime): > > Lars Eggert wrote: > >> Hi, > >> > >> every few days or so, my -STABLE NFS server (v3 and v4) gets > >> wedged > >> with a ton of messages about "nfsd server cache flooded, try to > >> increase nfsrc_floodlevel" in the log, and nfsstat shows TCPPeak > >> at > >> 16385. It requires a reboot to unwedge, restarting the server does > >> not help. > >> > >> The clients are (mostly) six -CURRENT nfsv4 boxes that netboot > >> from > >> the server and mount all drives from there. > >> Have you tried increasing vfs.nfsd.tcphighwater? This needs to be increased to increase the flood level above 16384. Garrett Wollman sets: vfs.nfsd.tcphighwater=3D100000 vfs.nfsd.tcpcachetimeo=3D300 or something like that, if I recall correctly. rick > >> I googled around and saw that others have hit this issue, but I > >> haven't seen any resolution posted. I guess I can increase > >> NFSRVCACHE_FLOODLEVEL in the source, but I wonder if I wouldn't > >> simply hit the increase value after a little while longer... > >> > >> Lars > >> > > You can either try this patch (which dynamically adjusts > > nfsrc_floodlevel > > along with handling a variety of overhead issues for the DRC under > > heavy load): > > http://people.freebsd.org/~rmacklem/drc4.patch > > > > or just bump it up a bunch. The default value was safe for a server > > with 256Mbytes > > of ram and a default mbuf cluster limit. The only thing you might > > have to do > > along with bumping NFSRC_FLOODLEVEL up is increasing > > kern.ipc.mbclusters. > > > > The variant of the above patch will make it into head someday, once > > I merge > > in changes from ivoras@'s similar patch and confer with him about > > it. >=20 > Dear all, >=20 > regarding the conversation from last year - quoted above, > I think I found the mentioned patch (it's variants) MFCd in r255532 > (from > http://svnweb.freebsd.org/base?view=3Drevision&revision=3D254337), > so it's included in 9.3-RELEASE. >=20 > Unfortunately I'm still having the nfsrc_floodlevel problem with > OpenOwner=3D16385, CacheSize=3D16385 (in nfsstat -e -s) in my production > environment under 9.3-RELEASE-amd64. > Extremely light load on the server (2 (FreeBSD8/9) clients), but the > building client (nfsv4) locks up frequently. It mounts 'home' and > 'ports/ports' via NFSv4 (this time, 'make index' in nfs-mounted > /usr/ports killed the nfsv4server). >=20 >=20 > I found another interesting 3 years old patch/thread, which seems > never beeing comitted: > http://lists.freebsd.org/pipermail/freebsd-fs/2011-July/012016.html >=20 > I don't really understand all these details of nfs(v4), but I observe > problems with regular usage, so I wanted to ask if there are new > findings regarding the "nfsd server cache flooded, try to increase > nfsrc_floodlevel" messages (while 'nfsrc_floodlevel' doesn't seem to > be > tunable in 9.3). > To my understanding, it's a problem on the server side, right? >=20 > Is the fix from 3 years back still adequate (does apply with view > offsets only to 9.3)? >=20 > I'm currently testing 9.3-RELEASE+noopen.patch, but it usually took > two > or three days until the client locked up (hadn't looked for the > reason > before the last issue, nfs(v4) was brand new reintroduced here) >=20 > Thanks, >=20 > -Harry >=20 >=20