From owner-freebsd-stable@FreeBSD.ORG Thu May 25 20:11:09 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2CCB716C09F for ; Thu, 25 May 2006 20:11:09 +0000 (UTC) (envelope-from grafan@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.188]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2FE3043D46 for ; Thu, 25 May 2006 20:11:07 +0000 (GMT) (envelope-from grafan@gmail.com) Received: by nf-out-0910.google.com with SMTP id k26so148067nfc for ; Thu, 25 May 2006 13:11:06 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=sN0v2BafHWlyuUeo+WiTLk3BxwVOAxNJVh7E/DJxX96gtuyc43+kmLX/yPd5vlUvY8sVrcZWaRsRPoE1RUwQJl4UnF5yrn4BZTQfMrrcB+fiA209q22S+XtY19QPhrBKtWlOjv2sUhAc0G5fnqMitHIjr7Fekd1OUaLE91MFO48= Received: by 10.49.7.18 with SMTP id k18mr6947891nfi; Thu, 25 May 2006 13:04:56 -0700 (PDT) Received: by 10.48.30.7 with HTTP; Thu, 25 May 2006 13:04:56 -0700 (PDT) Message-ID: <6eb82e0605251304y6cb57da5t99568bbbc19d1fa6@mail.gmail.com> Date: Thu, 25 May 2006 16:04:56 -0400 From: "Rong-en Fan" To: "Konstantin Belousov" In-Reply-To: <20060525145809.GP54541@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <001401c67f56$b02975e0$071872cf@Leadmon.local> <003001c67fae$27a88370$071872cf@Leadmon.local> <20060525051926.GB97976@xor.obsecurity.org> <20060525145809.GP54541@deviant.kiev.zoral.com.ua> Cc: freebsd-stable@freebsd.org, Howard Leadmon , Kris Kennaway Subject: Re: [patch, try 1] Re: Trouble with NFSd under 6.1-Stable, any ideas? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2006 20:11:27 -0000 On 5/25/06, Konstantin Belousov wrote: > On Thu, May 25, 2006 at 01:19:26AM -0400, Kris Kennaway wrote: > > On Wed, May 24, 2006 at 11:48:53PM -0400, Howard Leadmon wrote: > > > > > So what's changed at that delta, under the one that works vfs_lookup.= c is: > > > > > > Edit src/sys/kern/vfs_lookup.c > > > Add delta 1.80.2.6 2006.03.31.07.39.24 kris > > > > > > > > > Under the one that fails the vfs_lookup.c is: > > > > > > Edit src/sys/kern/vfs_lookup.c > > > Add delta 1.80.2.7 2006.04.30.03.57.46 kris > > > > > > > > > > > > So I stand corrected on my last post, the issue is in fact in this m= odule, as > > > just taking that module back to 1.80.2.6 fixes the problem with my se= rver. I > > > even took multiple NFS clients and gave them a heavy workload, and CP= U still > > > remained reasonable, and very responsive. As soon as I rev to the ne= w > > > version, NFS breaks badly and even a single client doing something li= ke a du > > > of a directory structure results in sluggishness and extreme CPU usag= e. > > > > Yep, unfortunately this commit was necessary to fix other bugs. Jeff > > said he should have time to look at it next week. > > > > Kris > > I tried to debug the problem. First, I have to admit that I cannot > reproduce the problem on GENERIC kernel. Only after QUOTAS where added, > and, correspondingly, UFS started to require Giant, > I get described behaviour. Below are the changes to GENERIC config file > I made to reproduce problem. > [...] > After that, server machine easily panics on > > KASSERT(!(debug_mpsafenet =3D=3D 1 && mtx_owned(&Giant)), > ("nfssvc_nfsd(): debug.mpsafenet=3D1 && Giant")); > > from nfsserver/nfs_syscalls.c, line 570. > > As I understand the problem, kern/vfs_lookup.c:lookup() could > aquire additional locks on Giant, indicating this by GIANTHELD > flag in nd. All processing in nfsserver already goes with Giant held, > so, I just dropped that excessive locks after return from lookup. > System with patch applied survived smoke test (client did > du on mounted dir, patch was generated from exported fs, etc.). > nfsd eats no more than 25% of CPU (with INVARIANTS). > > Please, users who reported the problem and willing to help, > try the patch (generated against STABLE) and give the feedback. > [...] Hi Konstantin and others, I'm now running RELENG_6_1 as of Apr 30 04:00 UTC source + your patch. The nfsd is quite happy! After client's du finishes, it stays idle as expected (eats 0.00% CPU). Thank you very much. Regards, Rong-En Fan