From owner-freebsd-fs@FreeBSD.ORG Wed May 16 17:28:37 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 108CD16A401 for ; Wed, 16 May 2007 17:28:37 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from gw.sandvine.com (gw.sandvine.com [199.243.201.138]) by mx1.freebsd.org (Postfix) with ESMTP id 4D89013C457 for ; Wed, 16 May 2007 17:28:36 +0000 (UTC) (envelope-from aedwards@sandvine.com) Received: from exchange-2.sandvine.com ([192.168.16.12]) by gw.sandvine.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 16 May 2007 13:16:29 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Date: Wed, 16 May 2007 13:16:27 -0400 Message-ID: <5230D3C40B842D4F9FB3CD368021BEF72F08F3@exchange-2.sandvine.com> In-Reply-To: <20070516163305.GA73495@xor.obsecurity.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Ufs dead-locks on freebsd 6.2 Thread-Index: AceX1+P2HSPmcMyKSLK15m7jv2eBpAAAVXqQ From: "Andrew Edwards" To: X-OriginalArrivalTime: 16 May 2007 17:16:29.0850 (UTC) FILETIME=[F17E47A0:01C797DD] Cc: freebsd-fs@freebsd.org Subject: RE: Ufs dead-locks on freebsd 6.2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 May 2007 17:28:37 -0000 Here's the backtrace from the last crash along with the output from show alllocks when the system was deadlocked. I have been running 6.2-release and compliled with makeoptions debug=3D-g, invariants, invariant_support and witness. I will update to 6-STABLE add diagnositc, debug_locks and debug_vfs_locks as per the handbook recommendation and retry. Yes, when the system was un-usable I was still able to ping it. I have the serial console setup as the default console so I can remotely access the box and break into the debugger etc. (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc059b480 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc059b795 in panic (fmt=3D0xc0787b04 "Most recently used by %s\n") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc06c4a4d in mtrash_ctor (mem=3D0xce74fa00, size=3D0, arg=3D0x0, flags=3D258) at /usr/src/sys/vm/uma_dbg.c:137 #4 0xc06c2c07 in uma_zalloc_arg (zone=3D0xc10615a0, udata=3D0x0, = flags=3D258) at /usr/src/sys/vm/uma_core.c:1850 #5 0xc0591416 in malloc (size=3D272, mtp=3D0xc07c32c0, flags=3D258) at uma.h:275 #6 0xc05edfab in __mnt_vnode_first (mvp=3D0xf3741c48, mp=3D0xcaa14cf8) at /usr/src/sys/kern/vfs_mount.c:1813 #7 0xc05f2467 in vfs_msync (mp=3D0xcaa14cf8, flags=3D2) at /usr/src/sys/kern/vfs_subr.c:2874 #8 0xc05f2bbd in sync_fsync (ap=3D0x0) at /usr/src/sys/kern/vfs_subr.c:3119 #9 0xc072f4ee in VOP_FSYNC_APV (vop=3D0x0, a=3D0xf3741cbc) at vnode_if.c:1020 #10 0xc05f097c in sync_vnode (bo=3D0xca854e90, td=3D0xca435000) at vnode_if.h:537 #11 0xc05f0bf1 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698 #12 0xc0587248 in fork_exit (callout=3D0xc05f0a04 , = arg=3D0x0, frame=3D0xf3741d38) at /usr/src/sys/kern/kern_fork.c:821 #13 0xc070712c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 db> show alllocks Process 36596 (sshd) thread 0xd1238c00 (102406) exclusive sleep mutex vm object (standard object) r =3D 0 (0xce2c87bc) locked @ /usr/src/sys/vm/vm_object.c:446 exclusive sx user map r =3D 0 (0xd128060c) locked @ /usr/src/sys/vm/vm_map.c:307 Process 887 (sshd) thread 0xca7d2000 (100056) exclusive sleep mutex vm object (standard object) r =3D 0 (0xcb713ad4) locked @ /usr/src/sys/vm/vm_fault.c:297 exclusive sx user map r =3D 0 (0xcaae4734) locked @ /usr/src/sys/vm/vm_map.c:3074 db> show lockedvnods Locked vnodes 0xcaa78660: tag ufs, type VREG usecount 2, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xc1046738 ref 0 pages 1596 lock type ufs: EXCL (count 1) by thread 0xca689c00 (pid 536) with 1 pending ino 494620, on dev amrd0s1d 0xcaa86110: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xca85f738 ref 0 pages 44 lock type ufs: EXCL (count 1) by thread 0xca7d2780 (pid 715) ino 494633, on dev amrd0s1d 0xcabe4110: tag ufs, type VDIR usecount 12, writecount 0, refcount 14 mountedhere 0 flags () v_object 0xcab28840 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xca9bbd80 (pid 14253) with 3 pending ino 423947, on dev amrd0s1d 0xcb437990: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb3c3dec ref 0 pages 4100 lock type ufs: EXCL (count 1) by thread 0xcaffac00 (pid 20868) ino 282640, on dev amrd0s1d 0xcb99e550: tag ufs, type VDIR usecount 2, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xcef979cc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xca9bb600 (pid 881) with 1 pending ino 423987, on dev amrd0s1d 0xcfc97dd0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcd4975ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb748900 (pid 2518) ino 424275, on dev amrd0s1d 0xccad9aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xcf0c539c ref 0 pages 5 lock type ufs: EXCL (count 1) by thread 0xca7d1c00 (pid 600) ino 188446, on dev amrd0s1d 0xccb0f110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb4609cc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbf480 (pid 11054) ino 424100, on dev amrd0s1d 0xcc501bb0: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xcc7d19cc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb76d600 (pid 13743) with 1 pending ino 424279, on dev amrd0s1d 0xcf96b220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf135c60 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbf900 (pid 29458) ino 424374, on dev amrd0s1d 0xcc5bbbb0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcdd5b318 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaad9900 (pid 50782) ino 424276, on dev amrd0s1d 0xcec1d000: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcd3d7108 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb76dc00 (pid 59514) ino 424500, on dev amrd0s1d 0xcebe5110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xccee95ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb650780 (pid 59975) ino 424509, on dev amrd0s1d 0xce0c1880: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xca8b64a4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb768a80 (pid 69466) ino 424555, on dev amrd0s1d 0xcf652110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf4a318c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff8600 (pid 75577) ino 424579, on dev amrd0s1d 0xce282550: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf261318 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd0235a80 (pid 81734) ino 424927, on dev amrd0s1d 0xcc1d4dd0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcd4a6630 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaccb900 (pid 81772) ino 424928, on dev amrd0s1d 0xcb820bb0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb251ad4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadaf480 (pid 84037) ino 424935, on dev amrd0s1d 0xced5aaa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb784210 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd0236000 (pid 202) ino 425039, on dev amrd0s1d 0xcbe45220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce55de70 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaad9a80 (pid 230) ino 425043, on dev amrd0s1d 0xcc098220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce4a9dec ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbfd80 (pid 9902) ino 425093, on dev amrd0s1d 0xcd585110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf8c1e70 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaccba80 (pid 24017) ino 425144, on dev amrd0s1d 0xceeac000: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb1225ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff5a80 (pid 24775) ino 425149, on dev amrd0s1d 0xcc549aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcab2d318 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff8a80 (pid 42358) ino 425227, on dev amrd0s1d 0xcc6f7000: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfd4139c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbf600 (pid 43117) ino 425230, on dev amrd0s1d 0xccc44bb0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfd18d68 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadafd80 (pid 42859) ino 425234, on dev amrd0s1d 0xcc7a7220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfedf420 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb76c600 (pid 48968) ino 425264, on dev amrd0s1d 0xcc693aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf92f738 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb655300 (pid 55381) ino 425286, on dev amrd0s1d 0xcbabf220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfe297bc ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd0234a80 (pid 63802) ino 425322, on dev amrd0s1d 0xcd760220: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcb11ac60 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadb0480 (pid 69938) ino 425348, on dev amrd0s1d 0xcc044990: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcaaff084 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadeda80 (pid 70418) ino 425360, on dev amrd0s1d 0xcc190660: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfed9108 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xca7d2900 (pid 76803) ino 425378, on dev amrd0s1d 0xcc676330: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf8c14a4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff8900 (pid 76841) ino 425384, on dev amrd0s1d 0xcf0ad110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce53b5ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadb0900 (pid 79849) ino 425394, on dev amrd0s1d 0xce4f6aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce3cc4a4 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb767300 (pid 79620) ino 425402, on dev amrd0s1d 0xce80d110: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf502108 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff5d80 (pid 98225) ino 425478, on dev amrd0s1d 0xcd218990: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf08e18c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff5000 (pid 98241) ino 425482, on dev amrd0s1d 0xcbcb3440: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf8a8738 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaff8180 (pid 1341) ino 425505, on dev amrd0s1d 0xcf6fe440: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf88f39c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcaad7900 (pid 4512) ino 425512, on dev amrd0s1d 0xcdd07aa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf7e9a50 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadaf600 (pid 4464) ino 425513, on dev amrd0s1d 0xcc18eaa0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xce2fa108 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcafbf300 (pid 13669) ino 425549, on dev amrd0s1d 0xcb9e1440: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcfbea39c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb64e480 (pid 13656) ino 425555, on dev amrd0s1d 0xcb8a8bb0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf6565ac ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcadb0180 (pid 22845) ino 425596, on dev amrd0s1d 0xccc47660: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcf551b58 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xcb64e180 (pid 22870) ino 425597, on dev amrd0s1d 0xcec8ecc0: tag ufs, type VREG usecount 1, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xcab61948 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd0234600 (pid 32036) ino 425633, on dev amrd0s1d 0xcdca1bb0: tag ufs, type VREG usecount 2, writecount 2, refcount 888 mountedhere 0 flags () v_object 0xcaf23294 ref 0 pages 41308 lock type ufs: EXCL (count 1) by thread 0xcadb0a80 (pid 5541) ino 130855246, on dev amrd1s1d=20 > -----Original Message----- > From: Kris Kennaway [mailto:kris@obsecurity.org]=20 > Sent: Wednesday, May 16, 2007 12:33 PM > To: Andrew Edwards > Cc: freebsd-performance@freebsd.org > Subject: Re: Ufs dead-locks on freebsd 6.2 >=20 > On Wed, May 16, 2007 at 12:08:24PM -0400, Andrew Edwards wrote: > > I have a system running a dual intel zeon 2.8Ghz with 4G of ram and=20 > > using an intel raid controller model SRCU42X which uses the=20 > amr driver. > > I have had this server running 5.4 upgraded to 6.2 and was running=20 > > fine for several months and then after a normal reboot I've started=20 > > having all sorts of problems with what appears to be=20 > dead-locks in the=20 > > filesystem. This server is my backup server and I rsync files from=20 > > various servers onto this one fairly non-stop. If I stop=20 > the rsync's=20 > > the system appears to be stable although I did have a=20 > kernel core just=20 > > last night. > >=20 > > When I have been able to observe the problem I ususally see one=20 > > filesystem become inaccessible, perhaps var but I'm not=20 > sure, and then=20 > > in a short period of time the whole system is inaccessible.=20 > Usually=20 > > if I startup just one of the rsync's within a couple of hours the=20 > > system will be un-usable. > >=20 > > I did find this thread which seems to describe similar=20 > issues but this=20 > > is a different driver. > >=20 > http://lists.freebsd.org/pipermail/freebsd-questions/2006-Augu st/127835. > > html >=20 > Probably not relevant then. Deadlocks come in many=20 > varieties, all different. >=20 > > Currently I'm running with debug.mpsafevfs=3D0, = debug.mpsafenet=3D1and=20 > > debug.mpsafevm=3D0 but this doesn't seem to help. > >=20 > > On perahps a related issue I have two other nearly=20 > identical systems=20 > > which were going to be upgrading to 6.2 as on 5.4 I am experiencing=20 > > deadlocks and when I hit ctrl-t I see the system is either stuck in=20 > > ufs or zoneinfo and I have not found very much information=20 > about zoneinfo. > >=20 > > Does anyone have any suggestions on what I can look for or other=20 > > tuning options or had similar experiences? >=20 > See the chapter on kernel debugging in the developers=20 > handbook for the information you need to provide before we=20 > can begin to debug your problem. >=20 > Kris >=20 >=20