From owner-freebsd-stable@FreeBSD.ORG Wed May 26 00:43:38 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA93C106566C for ; Wed, 26 May 2010 00:43:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 7C7628FC13 for ; Wed, 26 May 2010 00:43:38 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAOoL/EuDaFvG/2dsb2JhbACeGnHAMYUTBA X-IronPort-AV: E=Sophos;i="4.53,300,1272859200"; d="scan'208";a="77661518" Received: from amazon.cs.uoguelph.ca ([131.104.91.198]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 25 May 2010 20:43:36 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by amazon.cs.uoguelph.ca (Postfix) with ESMTP id 9549B2101C1; Tue, 25 May 2010 20:43:37 -0400 (EDT) X-Virus-Scanned: amavisd-new at amazon.cs.uoguelph.ca Received: from amazon.cs.uoguelph.ca ([127.0.0.1]) by localhost (amazon.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H3CQGnLvHixm; Tue, 25 May 2010 20:43:36 -0400 (EDT) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by amazon.cs.uoguelph.ca (Postfix) with ESMTP id 87D972101C0; Tue, 25 May 2010 20:43:36 -0400 (EDT) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o4Q0x8G24467; Tue, 25 May 2010 20:59:08 -0400 (EDT) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Tue, 25 May 2010 20:59:08 -0400 (EDT) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Mark Morley In-Reply-To: <20100525215230.CCC9C2101D5@amazon.cs.uoguelph.ca> Message-ID: References: <20100525215230.CCC9C2101D5@amazon.cs.uoguelph.ca> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-stable@freebsd.org Subject: Re: NFS trouble on 7.3-STABLE i386 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 May 2010 00:43:39 -0000 On Tue, 25 May 2010, Mark Morley wrote: > On Fri, 21 May 2010 11:32:33 -0400 (EDT) Rick Macklem wrote: On Fri, 21 May 2010, Mark Morley wrote: > >> Having an issue with a file server here (7.3-STABLE i386) >> >> The nfsd processes are hanging. Client access to the nfs shares stops working and the nfsd processes on the server cannot be killed by any means. There are no errors showing up anywhere on the server. The network connection to the server seems fine (ie: anything other than nfs traffic seems ok). Rebooting the server fixes the problem for a while, but it doesn't reboot easily. It times out on terminating the nfsd processes. When it finally does reboot the file system isn't marked clean, resulting in a long wait for fsck (although it doesn't find any problems, it's a multi terrabyte share and it takes a while). >> >> This morning it did it again. This time I tried manually killing nfsd but nothing I did would make them die. No errors. >> > Next time it happens, do a "ps axlH" to see what the nfsd threads are > waiting for. It might give you a hint as to what is happening. > > Ok, it did it again. ps axlH shows all the nfsd processes stuck in the _ufs_ state. The server isn't doing anything else, no other processes seem to be monopolizing resources or disks in any way. > > rpcinfo doesn't show anything amiss as far as I can tell (ie: rpc is running) > > After a reboot, one of the 32 nfsd's almost immediately goes into the "ufs" state and never leaves it (and never racks up and CPU time either). The others are fine. Slowly over time more and more enter this state. When I rebooted it today, all but one were in that state. The clients were bogging down, presumably because the one and only functioning nfsd was overworked. > You could try this patch. (It reverts the only vnode locking change that I can see was done the the nfs server between 7.1 and 7.3.): --- nfs_serv.c.sav 2010-05-25 19:40:29.000000000 -0400 +++ nfs_serv.c 2010-05-25 19:41:38.000000000 -0400 @@ -3236,7 +3236,7 @@ io.uio_rw = UIO_READ; io.uio_td = NULL; eofflag = 0; - vn_lock(vp, LK_SHARED | LK_RETRY, td); + vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, td); if (cookies) { free((caddr_t)cookies, M_TEMP); cookies = NULL; @@ -3518,7 +3518,7 @@ io.uio_rw = UIO_READ; io.uio_td = NULL; eofflag = 0; - vn_lock(vp, LK_SHARED | LK_RETRY, td); + vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, td); if (cookies) { free((caddr_t)cookies, M_TEMP); cookies = NULL; If you get a chance to try it, please let us know if it helps, rick