Date: Thu, 25 Feb 2010 01:05:02 -0800 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: freebsd-stable@freebsd.org Subject: Re: vfs deadlock during panic? Message-ID: <20100225090502.GA84181@icarus.home.lan> In-Reply-To: <D13CB108B048BD47B69C0CA1E0B5C03201113360@hq-es.FASTSOFT.COM> References: <D13CB108B048BD47B69C0CA1E0B5C0320111335C@hq-es.FASTSOFT.COM> <20100225073317.GA82327@icarus.home.lan> <D13CB108B048BD47B69C0CA1E0B5C03201113360@hq-es.FASTSOFT.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Feb 25, 2010 at 12:14:04AM -0800, Jake Holland wrote: > > All the backtraces indicate UFS snapshots are being used, as in > > you're doing dump -L or mksnap_ffs. Is that the case? > > Not to my knowledge, but I'm not sure of everything that's happening > during startup. The 3 I listed last time appear in the serial console, > usually within 10s of the login prompt's first appearance after reboot. > If I do more file operations, I typically see more such messages, but > not every time. > > For example, I tried running umount on my /var/crash (mounted on > /dev/mfid0s3d) and it gave me this: > > lock order reversal: > 1st 0xffffff00104337e8 ufs (ufs) @ kern/vfs_mount.c:1204 > 2nd 0xffffff001045aa58 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1194 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2e > witness_checkorder() at witness_checkorder+0x81e > __lockmgr_args() at __lockmgr_args+0xd31 > vop_stdlock() at vop_stdlock+0x39 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b > _vn_lock() at _vn_lock+0x57 > ffs_flushfiles() at ffs_flushfiles+0xc5 > softdep_flushfiles() at softdep_flushfiles+0x63 > ffs_unmount() at ffs_unmount+0x2e1 > dounmount() at dounmount+0x2e6 > unmount() at unmount+0x27e > syscall() at syscall+0x102 > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (22, FreeBSD ELF64, unmount), rip = 0x80069f4bc, rsp = > 0x7fffffffe2f > 8, rbp = 0 --- > > But I got nothing when I ran mount -a afterwards. > > > Then I ran vi on /etc/ports.conf and saw this: > > lock order reversal: > 1st 0xffffff81983bd5b8 bufwait (bufwait) @ kern/vfs_bio.c:2559 > 2nd 0xffffff000bd2ac00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2e > witness_checkorder() at witness_checkorder+0x81e > _sx_xlock() at _sx_xlock+0x55 > ufsdirhash_acquire() at ufsdirhash_acquire+0x44 > ufsdirhash_add() at ufsdirhash_add+0x19 > ufs_direnter() at ufs_direnter+0x88b > ufs_makeinode() at ufs_makeinode+0x2a7 > VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 > vn_open_cred() at vn_open_cred+0x482 > kern_openat() at kern_openat+0x179 > syscall() at syscall+0x102 > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (5, FreeBSD ELF64, open), rip = 0x800ce075c, rsp = > 0x7fffffffe0d8, r > bp = 0xf --- > > > But only the first time. Subsequently re-opening the same or another > few arbitrary text files with vi did not produce any repeats. And on > some occasions, nothing came up even on the first run of vi, though > usually the first vi does give a similar stack. Are you sure one of the filesystems on the disk isn't corrupt? There's been reports of this problem in the past, but AFAIR it doesn't manifest itself in this manner. Just something to consider: if you have another disk you can put in the box and install FreeBSD 8.x + upgrade to latest -STABLE on it, and it doesn't behave this way, then I'd say there's some underlying filesystem corruption of some kind. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100225090502.GA84181>