From owner-freebsd-stable@FreeBSD.ORG Thu May 25 03:49:16 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1422216A421 for ; Thu, 25 May 2006 03:49:16 +0000 (UTC) (envelope-from howard@leadmon.net) Received: from ibm.leadmon.net (ibm.leadmon.net [207.114.24.13]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0742043D58 for ; Thu, 25 May 2006 03:49:10 +0000 (GMT) (envelope-from howard@leadmon.net) Received: from gamer (gamer.leadmon.net [207.114.24.7]) (authenticated bits=0) by ibm.leadmon.net (8.13.6/8.13.6/LNSG+ORDB+SCOP+NJABL+SBL+DSBL+SORBS+CBL+RHSBL) with ESMTP id k4P3mw6M003999; Wed, 24 May 2006 23:49:02 -0400 (EDT) (envelope-from howard@leadmon.net) Authentication-Results: ibm.leadmon.net from=howard@leadmon.net; sender-id=pass; spf=pass X-SenderID: Sendmail Sender-ID Filter v0.2.12 ibm.leadmon.net k4P3mw6M003999 From: "Howard Leadmon" To: "'Howard Leadmon'" , "'Rong-en Fan'" Date: Wed, 24 May 2006 23:48:53 -0400 Organization: Leadmon Networking Message-ID: <003001c67fae$27a88370$071872cf@Leadmon.local> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869 In-Reply-To: <001401c67f56$b02975e0$071872cf@Leadmon.local> Thread-Index: AcZ+nFrMjh8kgqd0TJmSfurtuMDP1AAua14AABQ8pFA= X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0 (ibm.leadmon.net [207.114.24.13]); Wed, 24 May 2006 23:49:02 -0400 (EDT) X-Virus-Scanned: ClamAV version 0.88.2, clamav-milter version 0.88.2 on ibm.leadmon.net X-Virus-Status: Clean Cc: 'Konstantin Belousov' , freebsd-stable@freebsd.org, 'Kris Kennaway' Subject: RE: Trouble with NFSd under 6.1-Stable, any ideas? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2006 03:49:16 -0000 I need to follow up to the below, as I am not sure why the below test with the vfs_lookup.c didn't pan out the first time, but with my new found knowledge on cvs I was determined to regress the system till I found the smoking gun so to speak, which I have done. First let me say that instead of running RELENG_6_1 like Rong-en is, I am running the RELENG_6 tree that I know updates more often, but seems to work well for me. OK, so as I said above I started to regress the system a couple days at a time, till suddenly NFS stared working again, so I knew at that point it was a change that was made. So then I started to narrow the time range, till I got to the point that it broke. Sure enough under the RELENG_6 branch, this time was as follows: *default tag=RELENG_6 date=2006.04.30.03.57.00 (Works OK) *default tag=RELENG_6 date=2006.04.30.03.58.00 (Broken) So what's changed at that delta, under the one that works vfs_lookup.c is: Edit src/sys/kern/vfs_lookup.c Add delta 1.80.2.6 2006.03.31.07.39.24 kris Under the one that fails the vfs_lookup.c is: Edit src/sys/kern/vfs_lookup.c Add delta 1.80.2.7 2006.04.30.03.57.46 kris So I stand corrected on my last post, the issue is in fact in this module, as just taking that module back to 1.80.2.6 fixes the problem with my server. I even took multiple NFS clients and gave them a heavy workload, and CPU still remained reasonable, and very responsive. As soon as I rev to the new version, NFS breaks badly and even a single client doing something like a du of a directory structure results in sluggishness and extreme CPU usage. I am not a coder, so not sure why this module was changed, but unless there is some good reason why the changes were needed I would suspect it needs to be rolled back, or something fixed. So Rong-en Fan, I think you were dead on with your analysis that the issue is in fact inside the vfs_lookup.c module. I hope this helps... --- Howard Leadmon - howard@leadmon.net http://www.leadmon.net > -----Original Message----- > From: owner-freebsd-stable@freebsd.org > [mailto:owner-freebsd-stable@freebsd.org] On Behalf Of Howard Leadmon > Sent: Wednesday, May 24, 2006 1:23 PM > To: 'Rong-en Fan' > Cc: 'Konstantin Belousov'; freebsd-stable@freebsd.org > Subject: RE: Trouble with NFSd under 6.1-Stable, any ideas? > > > Hello Rong-en, > > > As an update, I did the below, and I still had the issue with > either version > of vfs_lookup.c compiled in and running. > > On the bright side, I didn't realize you could step through > the cvs by date, guess I just never paid attention. So I > just stepped back to 'tag=RELENG_6 date=2006.04.20.00.00.00' > on my server, rebuilt and violla nfs is now running > perfect. > > So backing out something has fixed my problem, now to figure > out just what it > was. As I don't know what has caused this, I have done > complete buildworlds > to make sure everything updates which takes a few hours. I > am going to > start moving the cvs date forward till I get the problem > back, once I nail this down a bit more, I'll let you know > what I come up with. > > > > --- > Howard Leadmon > http://www.leadmon.net > > > > > -----Original Message----- > > From: Rong-en Fan [mailto:grafan@gmail.com] > > Sent: Tuesday, May 23, 2006 3:09 PM > > To: Howard Leadmon > > Cc: freebsd-stable@freebsd.org > > Subject: Re: Trouble with NFSd under 6.1-Stable, any ideas? > > > > On 5/23/06, Howard Leadmon wrote: > > > > > > Hello Rong-en, > > > > > > Thanks for the info on getting the debugger configured, > > and on the serial > > > console. I will have to try and play with the serial > > console thing more, I > > > just tried putting in the flags and the damn thing hung, I > > had to boot > > > from CD and take the stuff back out. > > > > > > One thing you mention below that concerns me is that you > > have version 1.90 of > > > the vfs_lookup.c file. I just did a less on > > /usr/src/sys/kern/vfs_lookup.c > > > and I see the following: > > > > > > FreeBSD: src/sys/kern/vfs_lookup.c,v 1.80.2.7 2006/04/30 > > 03:57:46 kris > > > Exp > > > > > > > > > I even did a cvsup (I use cvsup2.FreeBSD.org) to make sure > > I had the > > > current stuff before rebuilding the kernel just now, and > > still I see the same thing. > > > Is something fishy going on here, or did you by chance > make a typo?? > > > > Sorry for the confusion. rev 1.90 is the number for -HEAD. > To back out > > this MFC'ed change for RELENG_6_1, please cvsup to > > RELENG_6_1 date=2006.04.30.03.57.00. Then you should see it is > > > > 1.80.2.6 2006/03/31 07:39:24 kris > > > > To verify the effect of this revision. Please run RELENG_6_1 with > > 2006.04.30.03.57.00 and 2006.04.30.04.00.00. > > > > Regards, > > Rong-En Fan > > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" >