From owner-freebsd-stable@FreeBSD.ORG Thu Feb 25 09:05:04 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 556841065670 for ; Thu, 25 Feb 2010 09:05:04 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.westchester.pa.mail.comcast.net (qmta13.westchester.pa.mail.comcast.net [76.96.59.243]) by mx1.freebsd.org (Postfix) with ESMTP id 045EF8FC0A for ; Thu, 25 Feb 2010 09:05:03 +0000 (UTC) Received: from omta01.westchester.pa.mail.comcast.net ([76.96.62.11]) by qmta13.westchester.pa.mail.comcast.net with comcast id m94q1d0070EZKEL5D954K8; Thu, 25 Feb 2010 09:05:04 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta01.westchester.pa.mail.comcast.net with comcast id m9531d00A3S48mS3M953t1; Thu, 25 Feb 2010 09:05:04 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 3D0FE1E301A; Thu, 25 Feb 2010 01:05:02 -0800 (PST) Date: Thu, 25 Feb 2010 01:05:02 -0800 From: Jeremy Chadwick To: freebsd-stable@freebsd.org Message-ID: <20100225090502.GA84181@icarus.home.lan> References: <20100225073317.GA82327@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Subject: Re: vfs deadlock during panic? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2010 09:05:04 -0000 On Thu, Feb 25, 2010 at 12:14:04AM -0800, Jake Holland wrote: > > All the backtraces indicate UFS snapshots are being used, as in > > you're doing dump -L or mksnap_ffs. Is that the case? > > Not to my knowledge, but I'm not sure of everything that's happening > during startup. The 3 I listed last time appear in the serial console, > usually within 10s of the login prompt's first appearance after reboot. > If I do more file operations, I typically see more such messages, but > not every time. > > For example, I tried running umount on my /var/crash (mounted on > /dev/mfid0s3d) and it gave me this: > > lock order reversal: > 1st 0xffffff00104337e8 ufs (ufs) @ kern/vfs_mount.c:1204 > 2nd 0xffffff001045aa58 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1194 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2e > witness_checkorder() at witness_checkorder+0x81e > __lockmgr_args() at __lockmgr_args+0xd31 > vop_stdlock() at vop_stdlock+0x39 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b > _vn_lock() at _vn_lock+0x57 > ffs_flushfiles() at ffs_flushfiles+0xc5 > softdep_flushfiles() at softdep_flushfiles+0x63 > ffs_unmount() at ffs_unmount+0x2e1 > dounmount() at dounmount+0x2e6 > unmount() at unmount+0x27e > syscall() at syscall+0x102 > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (22, FreeBSD ELF64, unmount), rip = 0x80069f4bc, rsp = > 0x7fffffffe2f > 8, rbp = 0 --- > > But I got nothing when I ran mount -a afterwards. > > > Then I ran vi on /etc/ports.conf and saw this: > > lock order reversal: > 1st 0xffffff81983bd5b8 bufwait (bufwait) @ kern/vfs_bio.c:2559 > 2nd 0xffffff000bd2ac00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2e > witness_checkorder() at witness_checkorder+0x81e > _sx_xlock() at _sx_xlock+0x55 > ufsdirhash_acquire() at ufsdirhash_acquire+0x44 > ufsdirhash_add() at ufsdirhash_add+0x19 > ufs_direnter() at ufs_direnter+0x88b > ufs_makeinode() at ufs_makeinode+0x2a7 > VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 > vn_open_cred() at vn_open_cred+0x482 > kern_openat() at kern_openat+0x179 > syscall() at syscall+0x102 > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (5, FreeBSD ELF64, open), rip = 0x800ce075c, rsp = > 0x7fffffffe0d8, r > bp = 0xf --- > > > But only the first time. Subsequently re-opening the same or another > few arbitrary text files with vi did not produce any repeats. And on > some occasions, nothing came up even on the first run of vi, though > usually the first vi does give a similar stack. Are you sure one of the filesystems on the disk isn't corrupt? There's been reports of this problem in the past, but AFAIR it doesn't manifest itself in this manner. Just something to consider: if you have another disk you can put in the box and install FreeBSD 8.x + upgrade to latest -STABLE on it, and it doesn't behave this way, then I'd say there's some underlying filesystem corruption of some kind. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |