From owner-freebsd-stable@FreeBSD.ORG Wed Jul 27 12:08:58 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7AA6106564A; Wed, 27 Jul 2011 12:08:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 39D668FC15; Wed, 27 Jul 2011 12:08:57 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p6RC8pj1048173 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 27 Jul 2011 15:08:51 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p6RC8pBu023999; Wed, 27 Jul 2011 15:08:51 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p6RC8oqN023998; Wed, 27 Jul 2011 15:08:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 27 Jul 2011 15:08:50 +0300 From: Kostik Belousov To: Rick Macklem Message-ID: <20110727120850.GT17489@deviant.kiev.zoral.com.ua> References: <20110726093258.GF17489@deviant.kiev.zoral.com.ua> <132828699.1045046.1311721943354.JavaMail.root@erie.cs.uoguelph.ca> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Ro3o5bxpeh65nwUH" Content-Disposition: inline In-Reply-To: <132828699.1045046.1311721943354.JavaMail.root@erie.cs.uoguelph.ca> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: rmacklem@freebsd.org, Herve Boulouis , freebsd-stable@freebsd.org Subject: Re: Sleeping thread owns a nonsleepable lock panic (& lor) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jul 2011 12:08:58 -0000 --Ro3o5bxpeh65nwUH Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 26, 2011 at 07:12:23PM -0400, Rick Macklem wrote: > Kostik Belousov wrote: > > On Tue, Jul 26, 2011 at 01:17:52PM +0200, Herve Boulouis wrote: > > > Le 26/07/2011 12:06, Kostik Belousov a =E9crit: > > > > On Tue, Jul 26, 2011 at 11:49:13AM +0200, Herve Boulouis wrote: > > > > > Le 25/07/2011 11:59, Kostik Belousov a ?crit: > > > > > > > > > > Ok the patched server crashed this morning strangely : all httpd > > > > > processes were stuck in nfs or vmopar > > > > > and were unkillable. Below is the full ps. > > > > > > > > Please see the > > > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handboo= k/kerneldebug-deadlocks.html > > > > for information required to debug the deadlocks. > > > > > > the box was not stricly deadlocked since I was able to interact with > > > it but I suppose you want me to > > > break into debugger when the symptoms appears again and report all > > > the commands listed in the handbook > > > deadlock section ? > >=20 > > Exactly. > >=20 > > I think everything was hung that accessed an nfs mount point. > > From the usermode, procstat -kk could catch some interesting > > information, > > but it is redundant if ddb output is captured. >=20 > Would it be worth considering reverting r223054? > (Note that I don't understand the VM side, so this may be completely > wrong:-) >=20 > The sleeps on vmopar could be happening because a dirty page is busy > and r223054 changes the VM_PAGER_xx value set a couple of ways. > 1 - When it returns VM_PAGER_ERROR instead of VM_PAGER_AGAIN, the > return value of "runlen" from vm_pageout_flush() changes. > 2 - I'm not sure, but I think the pre-r223054 code marked a partially > written page as VM_PAGER_OK instead of VM_PAGER_AGAIN? > (I'm wondering about this one, since the problem seems to happen > when the file's size has been truncated.) >=20 > Herve Boulouis, if you want to see what r223054 changes, just go to > http://svn.freebsd.org/viewvc/stable/8/sys/nfsclient > and then click on nfs_bio.c. > (The changes are small and could easily be reverted with a manual > edit.) >=20 > Since r223054 went into stable/8 on Jun 13, it seems a possible > explanation? rick I doubt it. The ps output makes it not very inplausible that the reporter got the LOR between vnode lock and page busy flag. The correct order is vnode lock -> busy bit. vmopar is a wait for the busy page state. Mentioned revision does not change the lock order. Anyway, this is only a speculation, until the requested data is provided. --Ro3o5bxpeh65nwUH Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk4v/9IACgkQC3+MBN1Mb4g1wwCdHVz5RdsVMC8sia2S5qw36Czi TH0Anj9y7UxYNzmvj80NZAdIxfvTNB20 =pguu -----END PGP SIGNATURE----- --Ro3o5bxpeh65nwUH--