Date: Thu, 17 May 2007 11:44:15 -0400 From: "Andrew Edwards" <aedwards@sandvine.com> To: <freebsd-performance@freebsd.org> Cc: freebsd-fs@freebsd.org Subject: RE: Ufs dead-locks on freebsd 6.2 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F0908@exchange-2.sandvine.com> In-Reply-To: <5230D3C40B842D4F9FB3CD368021BEF72F08F3@exchange-2.sandvine.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I've upgraded to 6-stable, added the kernel options as per the kernel
handbook. After about 5 hours of they system in a deadlock it panic'd.
Here's the backtrace, and show pcpu, show allpcpu, show locks, show
alllocks, show lockedvnods and alltrace.
I will have the system down for approx another 15-20mins if there's
anything else someone would like while I'm in the debugger.
db> bt
Tracing pid 46784 tid 105112 td 0xd44a8000
kdb_enter(c0785f13) at kdb_enter+0x2b
vfs_badlock(c0785f2c,c0786051,ccd47984) at vfs_badlock+0x47
assert_vop_locked(ccd47984,c0786051) at assert_vop_locked+0x4a
vop_lock_post(f9f709dc,0,1002,ccd47984,f9f709f8,...) at
vop_lock_post+0x2a
VOP_LOCK_APV(c07dc2e0,f9f709dc) at VOP_LOCK_APV+0xa0
vn_lock(ccd47984,1002,d44a8000) at vn_lock+0xac
lookup(f9f70c08) at lookup+0xde
namei(f9f70c08) at namei+0x39a
unp_connect(d44b2de8,d44dc380,d44a8000,d44b2de8,25,...) at
unp_connect+0xf0
uipc_connect(d44b2de8,d44dc380,d44a8000) at uipc_connect+0x66
soconnect(d44b2de8,d44dc380,d44a8000) at soconnect+0x4e
kern_connect(d44a8000,7,d44dc380,d44dc380,0,...) at kern_connect+0x74
connect(d44a8000,f9f70d04) at connect+0x2f
syscall(3b,805003b,bfbf003b,bfbfd920,bfbfd922,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (98, FreeBSD ELF32, connect), eip =3D 0x6812d553, esp =3D
0xbfbfd90c, ebp =3D 0xbfbfd9a8 ---
db> show pcpu
cpuid =3D 1
curthread =3D 0xd44a8000: pid 46784 "cron"
curpcb =3D 0xf9f70d90
fpcurthread =3D none
idlethread =3D 0xccb5c600: pid 10 "idle: cpu1"
APIC ID =3D 6
currentldt =3D 0x50
spin locks held:
db> show allpcpu
Current CPU: 1
cpuid =3D 0
curthread =3D 0xd4490d80: pid 46779 "cron"
curpcb =3D 0xf9f13d90
fpcurthread =3D none
idlethread =3D 0xccb5c780: pid 11 "idle: cpu0"
APIC ID =3D 0
currentldt =3D 0x50
spin locks held:
cpuid =3D 1
curthread =3D 0xd44a8000: pid 46784 "cron"
curpcb =3D 0xf9f70d90
fpcurthread =3D none
idlethread =3D 0xccb5c600: pid 10 "idle: cpu1"
APIC ID =3D 6
currentldt =3D 0x50
spin locks held:
db> show locks
db> show alllocks
db> show lockedvnods
Locked vnodes
0xd12d16cc: tag ufs, type VREG
usecount 1, writecount 1, refcount 4 mountedhere 0
flags ()
v_object 0xcd64ac60 ref 0 pages 48
lock type ufs: EXCL (count 1) by thread 0xcd6a6180 (pid 22025)#0
0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc06015c4 at vn_write+0x138
#5 0xc05c4544 at dofilewrite+0x7c
#6 0xc05c43e3 at kern_writev+0x3b
#7 0xc05c4309 at write+0x45
#8 0xc0723e2b at syscall+0x25b
#9 0xc070ee0f at Xint0x80_syscall+0x1f
ino 494646, on dev amrd0s1d
0xccf9f2b8: tag ufs, type VDIR
usecount 1, writecount 0, refcount 2495 mountedhere 0
flags (VV_ROOT)
v_object 0xd23c39cc ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xcd23fd80 (pid 40153) with
2493 pending#0 0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc05f5d12 at vget+0xbe
#5 0xc05ed9f9 at vfs_hash_get+0x8d
#6 0xc06b7b8f at ffs_vget+0x27
#7 0xc06c1435 at ufs_root+0x19
#8 0xc05eef1c at lookup+0x7c8
#9 0xc05ee4b2 at namei+0x39a
#10 0xc0600a13 at vn_open_cred+0x5b
#11 0xc06009b6 at vn_open+0x1e
#12 0xc05fa126 at kern_open+0xb6
#13 0xc05fa03a at open+0x1a
#14 0xc0723e2b at syscall+0x25b
#15 0xc070ee0f at Xint0x80_syscall+0x1f
ino 2, on dev amrd0s1d
0xcd030984: tag ufs, type VDIR
usecount 7, writecount 0, refcount 9 mountedhere 0
flags ()
v_object 0xcffb2108 ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xcd9bc780 (pid 29680) with
1 pending#0 0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc05f5d12 at vget+0xbe
#5 0xc05ea48e at cache_lookup+0x34a
#6 0xc05ea9c2 at vfs_cache_lookup+0x92
#7 0xc0737847 at VOP_LOOKUP_APV+0x87
#8 0xc05eebf8 at lookup+0x4a4
#9 0xc05ee4b2 at namei+0x39a
#10 0xc0600a13 at vn_open_cred+0x5b
#11 0xc06009b6 at vn_open+0x1e
#12 0xc05fa126 at kern_open+0xb6
#13 0xc05fa03a at open+0x1a
#14 0xc0723e2b at syscall+0x25b
#15 0xc070ee0f at Xint0x80_syscall+0x1f
ino 494592, on dev amrd0s1d
0xd1bad828: tag ufs, type VREG
usecount 2, writecount 2, refcount 6 mountedhere 0
flags ()
v_object 0xd08b28c4 ref 0 pages 82
lock type ufs: EXCL (count 1) by thread 0xcd5b0780 (pid 36375) with
2 pending#0 0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc06015c4 at vn_write+0x138
#5 0xc05c4544 at dofilewrite+0x7c
#6 0xc05c43e3 at kern_writev+0x3b
#7 0xc05c4309 at write+0x45
#8 0xc0723e2b at syscall+0x25b
#9 0xc070ee0f at Xint0x80_syscall+0x1f
ino 494775, on dev amrd0s1d
0xcfed0984: tag ufs, type VREG
usecount 1, writecount 1, refcount 4 mountedhere 0
flags ()
v_object 0xd0dcf39c ref 0 pages 7
lock type ufs: EXCL (count 1) by thread 0xccfa9000 (pid 653)#0
0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc05fd950 at fsync+0x9c
#5 0xc0723e2b at syscall+0x25b
#6 0xc070ee0f at Xint0x80_syscall+0x1f
ino 188447, on dev amrd0s1d
0xd2e3b828: tag ufs, type VREG
usecount 2, writecount 2, refcount 1612 mountedhere 0
flags ()
v_object 0xd2dee630 ref 0 pages 6436
lock type ufs: SHARED (count 1) with 1 pending#0 0xc0593b80 at
lockmgr+0x160
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc06012fe at vn_read+0x132
#5 0xc05c4269 at dofileread+0x85
#6 0xc05c4102 at kern_readv+0x36
#7 0xc05c402d at read+0x45
#8 0xc0723e2b at syscall+0x25b
#9 0xc070ee0f at Xint0x80_syscall+0x1f
ino 130807917, on dev amrd1s1d
0xd157f570: tag ufs, type VREG
usecount 1, writecount 1, refcount 2713 mountedhere 0
flags ()
v_object 0xd2d5118c ref 0 pages 245644
lock type ufs: EXCL (count 1) by thread 0xcd6a6300 (pid 26383)#0
0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc06015c4 at vn_write+0x138
#5 0xc05c4544 at dofilewrite+0x7c
#6 0xc05c43e3 at kern_writev+0x3b
#7 0xc05c4309 at write+0x45
#8 0xc0723e2b at syscall+0x25b
#9 0xc070ee0f at Xint0x80_syscall+0x1f
ino 116230408, on dev amrd1s1d
db> alltrace
Tracing command pid 46787 tid 105110 td 0xd44a8300
*** error reading from address 72d76109 ***
db>
> -----Original Message-----
> From: owner-freebsd-performance@freebsd.org=20
> [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of=20
> Andrew Edwards
> Sent: Wednesday, May 16, 2007 1:16 PM
> To: freebsd-performance@freebsd.org
> Cc: freebsd-fs@freebsd.org
> Subject: RE: Ufs dead-locks on freebsd 6.2
>=20
> Here's the backtrace from the last crash along with the=20
> output from show alllocks when the system was deadlocked. I=20
> have been running 6.2-release and compliled with makeoptions=20
> debug=3D-g, invariants, invariant_support and witness. I will=20
> update to 6-STABLE add diagnositc, debug_locks and=20
> debug_vfs_locks as per the handbook recommendation and retry.
>=20
> Yes, when the system was un-usable I was still able to ping=20
> it. I have the serial console setup as the default console=20
> so I can remotely access the box and break into the debugger etc.
>=20
> (kgdb) bt
> #0 doadump () at pcpu.h:165
> #1 0xc059b480 in boot (howto=3D260) at
> /usr/src/sys/kern/kern_shutdown.c:409
> #2 0xc059b795 in panic (fmt=3D0xc0787b04 "Most recently used by =
%s\n")
> at /usr/src/sys/kern/kern_shutdown.c:565
> #3 0xc06c4a4d in mtrash_ctor (mem=3D0xce74fa00, size=3D0, arg=3D0x0,
> flags=3D258)
> at /usr/src/sys/vm/uma_dbg.c:137
> #4 0xc06c2c07 in uma_zalloc_arg (zone=3D0xc10615a0, udata=3D0x0,=20
> flags=3D258)
> at /usr/src/sys/vm/uma_core.c:1850
> #5 0xc0591416 in malloc (size=3D272, mtp=3D0xc07c32c0, flags=3D258) =
at
> uma.h:275
> #6 0xc05edfab in __mnt_vnode_first (mvp=3D0xf3741c48, =
mp=3D0xcaa14cf8)
> at /usr/src/sys/kern/vfs_mount.c:1813
> #7 0xc05f2467 in vfs_msync (mp=3D0xcaa14cf8, flags=3D2)
> at /usr/src/sys/kern/vfs_subr.c:2874
> #8 0xc05f2bbd in sync_fsync (ap=3D0x0) at
> /usr/src/sys/kern/vfs_subr.c:3119
> #9 0xc072f4ee in VOP_FSYNC_APV (vop=3D0x0, a=3D0xf3741cbc) at=20
> vnode_if.c:1020 #10 0xc05f097c in sync_vnode (bo=3D0xca854e90,=20
> td=3D0xca435000) at
> vnode_if.h:537
> #11 0xc05f0bf1 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698
> #12 0xc0587248 in fork_exit (callout=3D0xc05f0a04 <sched_sync>, =
arg=3D0x0,
> frame=3D0xf3741d38) at /usr/src/sys/kern/kern_fork.c:821
> #13 0xc070712c in fork_trampoline () at
> /usr/src/sys/i386/i386/exception.s:208
>=20
>=20
> db> show alllocks
> Process 36596 (sshd) thread 0xd1238c00 (102406) exclusive=20
> sleep mutex vm object (standard object) r =3D 0 (0xce2c87bc)=20
> locked @ /usr/src/sys/vm/vm_object.c:446 exclusive sx user=20
> map r =3D 0 (0xd128060c) locked @
> /usr/src/sys/vm/vm_map.c:307
> Process 887 (sshd) thread 0xca7d2000 (100056) exclusive sleep=20
> mutex vm object (standard object) r =3D 0 (0xcb713ad4) locked @=20
> /usr/src/sys/vm/vm_fault.c:297 exclusive sx user map r =3D 0=20
> (0xcaae4734) locked @
> /usr/src/sys/vm/vm_map.c:3074
> db> show lockedvnods
> Locked vnodes
>=20
> 0xcaa78660: tag ufs, type VREG
> usecount 2, writecount 1, refcount 3 mountedhere 0
> flags ()
> v_object 0xc1046738 ref 0 pages 1596
> lock type ufs: EXCL (count 1) by thread 0xca689c00 (pid=20
> 536) with 1 pending
> ino 494620, on dev amrd0s1d
>=20
> 0xcaa86110: tag ufs, type VREG
> usecount 1, writecount 1, refcount 3 mountedhere 0
> flags ()
> v_object 0xca85f738 ref 0 pages 44
> lock type ufs: EXCL (count 1) by thread 0xca7d2780 (pid 715)
> ino 494633, on dev amrd0s1d
>=20
> 0xcabe4110: tag ufs, type VDIR
> usecount 12, writecount 0, refcount 14 mountedhere 0
> flags ()
> v_object 0xcab28840 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xca9bbd80 (pid=20
> 14253) with
> 3 pending
> ino 423947, on dev amrd0s1d
>=20
> 0xcb437990: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcb3c3dec ref 0 pages 4100
> lock type ufs: EXCL (count 1) by thread 0xcaffac00 (pid 20868)
> ino 282640, on dev amrd0s1d
>=20
> 0xcb99e550: tag ufs, type VDIR
> usecount 2, writecount 0, refcount 4 mountedhere 0
> flags ()
> v_object 0xcef979cc ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xca9bb600 (pid=20
> 881) with 1 pending
> ino 423987, on dev amrd0s1d
>=20
> 0xcfc97dd0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcd4975ac ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb748900 (pid 2518)
> ino 424275, on dev amrd0s1d
>=20
> 0xccad9aa0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 3 mountedhere 0
> flags ()
> v_object 0xcf0c539c ref 0 pages 5
> lock type ufs: EXCL (count 1) by thread 0xca7d1c00 (pid 600)
> ino 188446, on dev amrd0s1d
>=20
> 0xccb0f110: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcb4609cc ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcafbf480 (pid 11054)
> ino 424100, on dev amrd0s1d
>=20
> 0xcc501bb0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 3 mountedhere 0
> flags ()
> v_object 0xcc7d19cc ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb76d600 (pid=20
> 13743) with
> 1 pending
> ino 424279, on dev amrd0s1d
>=20
> 0xcf96b220: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf135c60 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcafbf900 (pid 29458)
> ino 424374, on dev amrd0s1d
>=20
> 0xcc5bbbb0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcdd5b318 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaad9900 (pid 50782)
> ino 424276, on dev amrd0s1d
>=20
> 0xcec1d000: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcd3d7108 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb76dc00 (pid 59514)
> ino 424500, on dev amrd0s1d
>=20
> 0xcebe5110: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xccee95ac ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb650780 (pid 59975)
> ino 424509, on dev amrd0s1d
>=20
> 0xce0c1880: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xca8b64a4 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb768a80 (pid 69466)
> ino 424555, on dev amrd0s1d
>=20
> 0xcf652110: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf4a318c ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaff8600 (pid 75577)
> ino 424579, on dev amrd0s1d
>=20
> 0xce282550: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf261318 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xd0235a80 (pid 81734)
> ino 424927, on dev amrd0s1d
>=20
> 0xcc1d4dd0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcd4a6630 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaccb900 (pid 81772)
> ino 424928, on dev amrd0s1d
>=20
> 0xcb820bb0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcb251ad4 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcadaf480 (pid 84037)
> ino 424935, on dev amrd0s1d
>=20
> 0xced5aaa0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcb784210 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xd0236000 (pid 202)
> ino 425039, on dev amrd0s1d
>=20
> 0xcbe45220: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xce55de70 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaad9a80 (pid 230)
> ino 425043, on dev amrd0s1d
>=20
> 0xcc098220: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xce4a9dec ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcafbfd80 (pid 9902)
> ino 425093, on dev amrd0s1d
>=20
> 0xcd585110: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf8c1e70 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaccba80 (pid 24017)
> ino 425144, on dev amrd0s1d
>=20
> 0xceeac000: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcb1225ac ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaff5a80 (pid 24775)
> ino 425149, on dev amrd0s1d
>=20
> 0xcc549aa0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcab2d318 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaff8a80 (pid 42358)
> ino 425227, on dev amrd0s1d
>=20
> 0xcc6f7000: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcfd4139c ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcafbf600 (pid 43117)
> ino 425230, on dev amrd0s1d
>=20
> 0xccc44bb0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcfd18d68 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcadafd80 (pid 42859)
> ino 425234, on dev amrd0s1d
>=20
> 0xcc7a7220: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcfedf420 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb76c600 (pid 48968)
> ino 425264, on dev amrd0s1d
>=20
> 0xcc693aa0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf92f738 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb655300 (pid 55381)
> ino 425286, on dev amrd0s1d
>=20
> 0xcbabf220: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcfe297bc ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xd0234a80 (pid 63802)
> ino 425322, on dev amrd0s1d
>=20
> 0xcd760220: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcb11ac60 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcadb0480 (pid 69938)
> ino 425348, on dev amrd0s1d
>=20
> 0xcc044990: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcaaff084 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcadeda80 (pid 70418)
> ino 425360, on dev amrd0s1d
>=20
> 0xcc190660: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcfed9108 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xca7d2900 (pid 76803)
> ino 425378, on dev amrd0s1d
>=20
> 0xcc676330: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf8c14a4 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaff8900 (pid 76841)
> ino 425384, on dev amrd0s1d
>=20
> 0xcf0ad110: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xce53b5ac ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcadb0900 (pid 79849)
> ino 425394, on dev amrd0s1d
>=20
> 0xce4f6aa0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xce3cc4a4 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb767300 (pid 79620)
> ino 425402, on dev amrd0s1d
>=20
> 0xce80d110: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf502108 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaff5d80 (pid 98225)
> ino 425478, on dev amrd0s1d
>=20
> 0xcd218990: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf08e18c ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaff5000 (pid 98241)
> ino 425482, on dev amrd0s1d
>=20
> 0xcbcb3440: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf8a8738 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaff8180 (pid 1341)
> ino 425505, on dev amrd0s1d
>=20
> 0xcf6fe440: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf88f39c ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcaad7900 (pid 4512)
> ino 425512, on dev amrd0s1d
>=20
> 0xcdd07aa0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf7e9a50 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcadaf600 (pid 4464)
> ino 425513, on dev amrd0s1d
>=20
> 0xcc18eaa0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xce2fa108 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcafbf300 (pid 13669)
> ino 425549, on dev amrd0s1d
>=20
> 0xcb9e1440: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcfbea39c ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb64e480 (pid 13656)
> ino 425555, on dev amrd0s1d
>=20
> 0xcb8a8bb0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf6565ac ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcadb0180 (pid 22845)
> ino 425596, on dev amrd0s1d
>=20
> 0xccc47660: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcf551b58 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xcb64e180 (pid 22870)
> ino 425597, on dev amrd0s1d
>=20
> 0xcec8ecc0: tag ufs, type VREG
> usecount 1, writecount 1, refcount 2 mountedhere 0
> flags ()
> v_object 0xcab61948 ref 0 pages 1
> lock type ufs: EXCL (count 1) by thread 0xd0234600 (pid 32036)
> ino 425633, on dev amrd0s1d
>=20
> 0xcdca1bb0: tag ufs, type VREG
> usecount 2, writecount 2, refcount 888 mountedhere 0
> flags ()
> v_object 0xcaf23294 ref 0 pages 41308
> lock type ufs: EXCL (count 1) by thread 0xcadb0a80 (pid 5541)
> ino 130855246, on dev amrd1s1d=20
>=20
> > -----Original Message-----
> > From: Kris Kennaway [mailto:kris@obsecurity.org]
> > Sent: Wednesday, May 16, 2007 12:33 PM
> > To: Andrew Edwards
> > Cc: freebsd-performance@freebsd.org
> > Subject: Re: Ufs dead-locks on freebsd 6.2
> >=20
> > On Wed, May 16, 2007 at 12:08:24PM -0400, Andrew Edwards wrote:
> > > I have a system running a dual intel zeon 2.8Ghz with 4G=20
> of ram and=20
> > > using an intel raid controller model SRCU42X which uses the
> > amr driver.
> > > I have had this server running 5.4 upgraded to 6.2 and=20
> was running=20
> > > fine for several months and then after a normal reboot=20
> I've started=20
> > > having all sorts of problems with what appears to be
> > dead-locks in the
> > > filesystem. This server is my backup server and I rsync=20
> files from=20
> > > various servers onto this one fairly non-stop. If I stop
> > the rsync's
> > > the system appears to be stable although I did have a
> > kernel core just
> > > last night.
> > >=20
> > > When I have been able to observe the problem I ususally see one=20
> > > filesystem become inaccessible, perhaps var but I'm not
> > sure, and then
> > > in a short period of time the whole system is inaccessible.=20
> > Usually
> > > if I startup just one of the rsync's within a couple of hours the=20
> > > system will be un-usable.
> > >=20
> > > I did find this thread which seems to describe similar
> > issues but this
> > > is a different driver.
> > >=20
> > http://lists.freebsd.org/pipermail/freebsd-questions/2006-Augu
> st/127835.
> > > html
> >=20
> > Probably not relevant then. Deadlocks come in many varieties, all=20
> > different.
> >=20
> > > Currently I'm running with debug.mpsafevfs=3D0,=20
> debug.mpsafenet=3D1and=20
> > > debug.mpsafevm=3D0 but this doesn't seem to help.
> > >=20
> > > On perahps a related issue I have two other nearly
> > identical systems
> > > which were going to be upgrading to 6.2 as on 5.4 I am=20
> experiencing=20
> > > deadlocks and when I hit ctrl-t I see the system is=20
> either stuck in=20
> > > ufs or zoneinfo and I have not found very much information
> > about zoneinfo.
> > >=20
> > > Does anyone have any suggestions on what I can look for or other=20
> > > tuning options or had similar experiences?
> >=20
> > See the chapter on kernel debugging in the developers=20
> handbook for the=20
> > information you need to provide before we can begin to debug your=20
> > problem.
> >=20
> > Kris
> >=20
> >=20
> _______________________________________________
> freebsd-performance@freebsd.org mailing list=20
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to=20
> "freebsd-performance-unsubscribe@freebsd.org"
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5230D3C40B842D4F9FB3CD368021BEF72F0908>