From owner-freebsd-current@FreeBSD.ORG Thu May 10 10:21:29 2012 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 332DB106564A; Thu, 10 May 2012 10:21:29 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by mx1.freebsd.org (Postfix) with ESMTP id 560318FC14; Thu, 10 May 2012 10:21:28 +0000 (UTC) Received: by wibhr2 with SMTP id hr2so241137wib.13 for ; Thu, 10 May 2012 03:21:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=v0q58T1qmX8kAUhIlKZNFQOgMoiUAJv/ZeU/6Q5Qeh0=; b=wAFVpzEvO1MNpACKBjC0K03ddJyk70jXTV09CTtRCl0SPcUqj/SMRm7zGSaND05OVX oX6a4BaHBTmlb5tLUc/F1eCranS/dIXHj9J6UY6mMJZbwytPMY6k3zT1K4ETaT+ncNqE 63jcWp8RrNHMzCG3O+pAg//fu8g3mPcMzgMAhlVJ8TbG5Ze/PMekLT3sDfCLLcTrcE3e HKecfAL0Ucn8Ksny9isPE/gZ1JT56dM2XW4pI9A8m4j3I1iV3kf2uXNf0SfOZJ+7ySwy WWYZjtCoLIVnCXfdZlxzeP9ewbIXXcjCG7sccjSCy+y5QJpuaU7lJTwkkPBhSPzYRbIk 80AA== Received: by 10.180.109.197 with SMTP id hu5mr8208824wib.8.1336645287204; Thu, 10 May 2012 03:21:27 -0700 (PDT) Received: from dft-labs.eu (dft-labs.eu. [80.87.128.179]) by mx.google.com with ESMTPS id m4sm2332741wix.0.2012.05.10.03.21.24 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 10 May 2012 03:21:25 -0700 (PDT) Date: Thu, 10 May 2012 12:21:18 +0200 From: Mateusz Guzik To: Peter Holm Message-ID: <20120510102118.GA26472@dft-labs.eu> References: <4FA6F324.4080107@FreeBSD.org> <4FA82269.6080406@FreeBSD.org> <20120507201153.GA19942@dft-labs.eu> <20120508194514.GA10688@x2.osted.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20120508194514.GA10688@x2.osted.lan> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: Doug Barton , Sergey Kandaurov , freebsd-current , mckusick@freebsd.org Subject: Re: panic, seems related to r234386 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 May 2012 10:21:29 -0000 On Tue, May 08, 2012 at 09:45:14PM +0200, Peter Holm wrote: > On Mon, May 07, 2012 at 10:11:53PM +0200, Mateusz Guzik wrote: > > On Mon, May 07, 2012 at 12:28:41PM -0700, Doug Barton wrote: > > > On 05/06/2012 15:19, Sergey Kandaurov wrote: > > > > On 7 May 2012 01:54, Doug Barton wrote: > > > >> I got this with today's current, previous (working) kernel is r232719. > > > >> > > > >> panic: _mtx_lock_sleep: recursed on non-recursive mutex struct mount mtx > > > >> @ /frontier/svn/head/sys/kern/vfs_subr.c:4595 > > > > > > ... > > > > > > > Please try this patch. > > > > > > > > Index: fs/ext2fs/ext2_vfsops.c > > > > =================================================================== > > > > --- fs/ext2fs/ext2_vfsops.c (revision 235108) > > > > +++ fs/ext2fs/ext2_vfsops.c (working copy) > > > > @@ -830,7 +830,6 @@ > > > > /* > > > > * Write back each (modified) inode. > > > > */ > > > > - MNT_ILOCK(mp); > > > > loop: > > > > MNT_VNODE_FOREACH_ALL(vp, mp, mvp) { > > > > if (vp->v_type == VNON) { > > > > > > > > > > Didn't help, sorry. I put 234385 through some pretty heavy load > > > yesterday, and everything was fine. As soon as I move up to 234386, the > > > panic triggered again. So I cleaned everything up, applied your patch, > > > built a kernel from scratch, and rebooted. It was Ok for a few seconds > > > after boot, then panic'ed again, I think in a different place, but I'm > > > not sure because subsequent attempts to fsck the file systems caused new > > > panics which overwrote the old ones before they could be saved. > > > > > > > Another MNT_ILOCK was hiding few lines below, try this patch: > > > > http://student.agh.edu.pl/~mjguzik/patches/ext2fs-ilock.patch > > > > I've tested this a bit and I believe this fixes your problem. > > > > Gave this a spin and found what looks like a deadlock: > > http://people.freebsd.org/~pho/stress/log/ext2fs.txt > > Not a new problem, it would seem. Same issue with 8.3-PRERELEASE r232656M. > pid 2680 (fts) holds lock for vnode cb4be414 and tries to lock cc0ac15c pid 2581 (openat) holds lock for vnode cc0ac15c and tries to lock cb4be414 openat calls rmdir foo/bar and ext2_rmdir unlocks and tries to lock again foo's vnode. This is fairly easly reproducible with concurrently running mkdir and fts testcase programs that are provided by stress2. I'll try to come up with a patch by the end of the week. -- Mateusz Guzik