From owner-freebsd-stable@FreeBSD.ORG Thu May 25 14:58:18 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7A4D216A474 for ; Thu, 25 May 2006 14:58:18 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from fw.zoral.com.ua (ll-227.216.82.212.sovam.net.ua [212.82.216.227]) by mx1.FreeBSD.org (Postfix) with ESMTP id 85AB143D53 for ; Thu, 25 May 2006 14:58:17 +0000 (GMT) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id k4PEwBBq047545 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 25 May 2006 17:58:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.6/8.13.6) with ESMTP id k4PEwBm2060351; Thu, 25 May 2006 17:58:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.6/8.13.6/Submit) id k4PEw9Js060350; Thu, 25 May 2006 17:58:09 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 25 May 2006 17:58:09 +0300 From: Konstantin Belousov To: Kris Kennaway Message-ID: <20060525145809.GP54541@deviant.kiev.zoral.com.ua> References: <001401c67f56$b02975e0$071872cf@Leadmon.local> <003001c67fae$27a88370$071872cf@Leadmon.local> <20060525051926.GB97976@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bJ3jXuwtxrXxD2iT" Content-Disposition: inline In-Reply-To: <20060525051926.GB97976@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i X-Virus-Scanned: ClamAV version 0.88.2, clamav-milter version 0.88.2 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on fw.zoral.com.ua Cc: 'Rong-en Fan' , Howard Leadmon , freebsd-stable@freebsd.org Subject: [patch, try 1] Re: Trouble with NFSd under 6.1-Stable, any ideas? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2006 14:58:18 -0000 --bJ3jXuwtxrXxD2iT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 25, 2006 at 01:19:26AM -0400, Kris Kennaway wrote: > On Wed, May 24, 2006 at 11:48:53PM -0400, Howard Leadmon wrote: >=20 > > So what's changed at that delta, under the one that works vfs_lookup.c = is: > >=20 > > Edit src/sys/kern/vfs_lookup.c > > Add delta 1.80.2.6 2006.03.31.07.39.24 kris > >=20 > >=20 > > Under the one that fails the vfs_lookup.c is: > >=20 > > Edit src/sys/kern/vfs_lookup.c > > Add delta 1.80.2.7 2006.04.30.03.57.46 kris > >=20 > >=20 > >=20 > > So I stand corrected on my last post, the issue is in fact in this mod= ule, as > > just taking that module back to 1.80.2.6 fixes the problem with my serv= er. I > > even took multiple NFS clients and gave them a heavy workload, and CPU = still > > remained reasonable, and very responsive. As soon as I rev to the new > > version, NFS breaks badly and even a single client doing something like= a du > > of a directory structure results in sluggishness and extreme CPU usage. >=20 > Yep, unfortunately this commit was necessary to fix other bugs. Jeff > said he should have time to look at it next week. >=20 > Kris I tried to debug the problem. First, I have to admit that I cannot reproduce the problem on GENERIC kernel. Only after QUOTAS where added, and, correspondingly, UFS started to require Giant, I get described behaviour. Below are the changes to GENERIC config file I made to reproduce problem. Index: amd64/conf/GENERIC =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /usr/local/arch/ncvs/src/sys/amd64/conf/GENERIC,v retrieving revision 1.439.2.11 diff -u -r1.439.2.11 GENERIC --- amd64/conf/GENERIC 30 Apr 2006 17:39:43 -0000 1.439.2.11 +++ amd64/conf/GENERIC 25 May 2006 14:44:14 -0000 @@ -26,6 +26,19 @@ #hints "GENERIC.hints" # Default places to look for devices. =20 makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols +options KDB +options KDB_TRACE +#options KDB_UNATTENDED +options DDB +options DDB_NUMSYM +options BREAK_TO_DEBUGGER +options INVARIANTS +options INVARIANT_SUPPORT +options WITNESS +options DEBUG_LOCKS +options DEBUG_VFS_LOCKS +options DIAGNOSTIC +options MUTEX_PROFILING =20 #options SCHED_ULE # ULE scheduler options SCHED_4BSD # 4BSD scheduler @@ -34,6 +47,7 @@ options INET6 # IPv6 communications protocols options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support +options QUOTA options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device After that, server machine easily panics on=20 KASSERT(!(debug_mpsafenet =3D=3D 1 && mtx_owned(&Giant)), ("nfssvc_nfsd(): debug.mpsafenet=3D1 && Giant")); from nfsserver/nfs_syscalls.c, line 570. As I understand the problem, kern/vfs_lookup.c:lookup() could aquire additional locks on Giant, indicating this by GIANTHELD flag in nd. All processing in nfsserver already goes with Giant held, so, I just dropped that excessive locks after return from lookup. System with patch applied survived smoke test (client did du on mounted dir, patch was generated from exported fs, etc.). nfsd eats no more than 25% of CPU (with INVARIANTS). Please, users who reported the problem and willing to help, try the patch (generated against STABLE) and give the feedback. Index: nfsserver/nfs_serv.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /usr/local/arch/ncvs/src/sys/nfsserver/nfs_serv.c,v retrieving revision 1.156.2.2 diff -u -r1.156.2.2 nfs_serv.c --- nfsserver/nfs_serv.c 13 Mar 2006 03:06:49 -0000 1.156.2.2 +++ nfsserver/nfs_serv.c 25 May 2006 14:44:25 -0000 @@ -569,6 +569,10 @@ =20 error =3D lookup(&ind); ind.ni_dvp =3D NULL; + if (ind.ni_cnd.cn_flags & GIANTHELD) { + mtx_unlock(&Giant); + ind.ni_cnd.cn_flags &=3D ~GIANTHELD; + } =20 if (error =3D=3D 0) { /* @@ -1915,6 +1919,10 @@ =20 error =3D lookup(&nd); nd.ni_dvp =3D NULL; + if (nd.ni_cnd.cn_flags & GIANTHELD) { + mtx_unlock(&Giant); + nd.ni_cnd.cn_flags &=3D ~GIANTHELD; + } if (error) goto ereply; =20 @@ -2141,6 +2149,10 @@ =20 error =3D lookup(&nd); nd.ni_dvp =3D NULL; + if (nd.ni_cnd.cn_flags & GIANTHELD) { + mtx_unlock(&Giant); + nd.ni_cnd.cn_flags &=3D ~GIANTHELD; + } =20 if (error) goto out; @@ -2878,6 +2890,10 @@ =20 error =3D lookup(&nd); nd.ni_dvp =3D NULL; + if (nd.ni_cnd.cn_flags & GIANTHELD) { + mtx_unlock(&Giant); + nd.ni_cnd.cn_flags &=3D ~GIANTHELD; + } =20 if (error =3D=3D 0) { bzero((caddr_t)fhp, sizeof(nfh)); Index: nfsserver/nfs_srvsubs.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /usr/local/arch/ncvs/src/sys/nfsserver/nfs_srvsubs.c,v retrieving revision 1.136.2.2 diff -u -r1.136.2.2 nfs_srvsubs.c --- nfsserver/nfs_srvsubs.c 4 Apr 2006 15:29:51 -0000 1.136.2.2 +++ nfsserver/nfs_srvsubs.c 25 May 2006 14:44:25 -0000 @@ -875,6 +875,10 @@ } if (!lockleaf) cnp->cn_flags &=3D ~LOCKLEAF; + if (cnp->cn_flags & GIANTHELD) { + mtx_unlock(&Giant); + cnp->cn_flags &=3D ~GIANTHELD; + } =20 /* * nfs_namei() guarentees that fields will not contain garbage --bJ3jXuwtxrXxD2iT Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEdcYAC3+MBN1Mb4gRAsSgAKDRwtPXPk/SzTdAYa4ZZ2cfKoN+BQCgi2yZ yA8iFmWWaxPDTuhH7AriVk0= =pCcP -----END PGP SIGNATURE----- --bJ3jXuwtxrXxD2iT--