From owner-freebsd-stable@FreeBSD.ORG Sat Jul 26 10:19:34 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ADDC03DC for ; Sat, 26 Jul 2014 10:19:34 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 07A972CBB for ; Sat, 26 Jul 2014 10:19:33 +0000 (UTC) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id s6QAJVo5083203; Sat, 26 Jul 2014 12:19:31 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 423CE3A6B; Sat, 26 Jul 2014 12:19:31 +0200 (CEST) Message-ID: <53D380B2.2080700@omnilan.de> Date: Sat, 26 Jul 2014 12:19:30 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: panic/lock on 9.3-RELEASE with nullfs/nfs/zfs combination References: <53D12973.3010805@omnilan.de> <20140724165917.GT93733@kib.kiev.ua> <53D1503B.2030200@omnilan.de> <20140724193048.GU93733@kib.kiev.ua> <53D2006C.7090207@omnilan.de> <20140725151448.GY93733@kib.kiev.ua> In-Reply-To: <20140725151448.GY93733@kib.kiev.ua> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig38F03A934E680565AC552C34" X-Greylist: ACL 119 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Sat, 26 Jul 2014 12:19:32 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) Cc: FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 10:19:34 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig38F03A934E680565AC552C34 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Bez=C3=BCglich Konstantin Belousov's Nachricht vom 25.07.2014 17:14 (localtime): > On Fri, Jul 25, 2014 at 08:59:56AM +0200, Harald Schmalzbauer wrote: >> =E2=80=A6 >> But there's another WITNESS message: >> WITNESS: unable to allocate a new witness object. >> Ill ask the relatives what they know about this message (aunt bing, >> uncle google aso). > Yes, this is the problem. In sys/kern/subr_witness.c, change the > #define WITNESS_COUNT from 1024 to e.g. 2048. Thank you very much, havn't found it myself :-( FreeBSD/amd64 (mira.inop.mo1.omnilan.net) (ttyu0) login: shared lock of (lockmgr) zfs @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/modules/zfs/../../cddl/c= ontrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1513 while exclusively locked from /usr/local/share/deploy-tools/RELENG_9_3/src/sys/modules/zfs/../../cddl/c= ontrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1511 panic: share->excl cpuid =3D 2 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff82e526da80 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e526db40 panic() at panic+0x1cd/frame 0xffffff82e526dc40 witness_checkorder() at witness_checkorder+0x161/frame 0xffffff82e526dd00= __lockmgr_args() at __lockmgr_args+0x150a/frame 0xffffff82e526dde0 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff82e526de00 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe3/frame 0xffffff82e526de30 _vn_lock() at _vn_lock+0x57/frame 0xffffff82e526de90 zfs_lookup() at zfs_lookup+0x420/frame 0xffffff82e526df20 zfs_freebsd_lookup() at zfs_freebsd_lookup+0xa6/frame 0xffffff82e526e070 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x102/frame 0xffffff82e526e0a0 vfs_cache_lookup() at vfs_cache_lookup+0xff/frame 0xffffff82e526e110 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x102/frame 0xffffff82e526e140 null_lookup() at null_lookup+0x92/frame 0xffffff82e526e1c0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x102/frame 0xffffff82e526e1f0 lookup() at lookup+0x32f/frame 0xffffff82e526e290 namei() at namei+0x3df/frame 0xffffff82e526e340 vn_open_cred() at vn_open_cred+0x1e2/frame 0xffffff82e526e4b0 vop_stdvptocnp() at vop_stdvptocnp+0x1af/frame 0xffffff82e526e7e0 null_vptocnp() at null_vptocnp+0xf5/frame 0xffffff82e526e850 VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x105/frame 0xffffff82e526e880 vn_vptocnp_locked() at vn_vptocnp_locked+0x15b/frame 0xffffff82e526e910 vn_fullpath1() at vn_fullpath1+0x100/frame 0xffffff82e526e970 kern___getcwd() at kern___getcwd+0xd4/frame 0xffffff82e526e9d0 amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e526eaf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e526eaf0 --- syscall (326, FreeBSD ELF64, sys___getcwd), rip =3D 0x8011a191c, rsp = =3D 0x7fffffffe658, rbp =3D 0x801873400 --- KDB: enter: panic [ thread pid 1668 tid 100854 ] Stopped at kdb_enter+0x3b: movq $0,0x645622(%rip) db> show alllocks Process 1668 (vim) thread 0xfffffe0033498920 (100854) exclusive lockmgr zfs (zfs) r =3D 0 (0xfffffe001dbc5488) locked @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/modules/zfs/../../cddl/c= ontrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1511 db> show mount =E2=80=A6 0xfffffe0033a24338 /zfs/netshares/deployment/pub/FreeBSD/OmniLAN/ports/ports on /.JAILfbsm/usr/ports (nullfs) =E2=80=A6 db> show mount 0xfffffe0033a24338 0xfffffe0033a24338 /zfs/netshares/deployment/pub/FreeBSD/OmniLAN/ports/ports on /.JAILfbsm/usr/ports (nullfs) mnt_flag =3D LOCAL mnt_kern_flag =3D EXTENDED_SHARED, SHARED_WRITES, LOOKUP_EXCL_DOTDOT, MPSAFE, LOOKUP_SHARED mnt_opt =3D rw, fstype, fspath, target, errmsg mnt_stat =3D { version=3D537068824 type=3D222 flags=3D0x0000000000001000 bsize=3D512 iosize=3D131072 blocks=3D6314029472 bfree=3D6309830941 bavail=3D6309830941 files=3D6310126321 ffree=3D6309830941 syncwrites=3D0 asyncwrites=3D0 syncreads=3D0 asyncreads=3D0 namemax=3D255 owner=3D0 fsid=3D[687931140, 41] } mnt_cred =3D { uid=3D0 ruid=3D0 } mnt_ref =3D 24 mnt_gen =3D 1 mnt_nvnodelistsize =3D 24 mnt_activevnodelistsize =3D 5 mnt_writeopcount =3D 0 mnt_maxsymlinklen =3D 0 mnt_iosize_max =3D 65536 mnt_hashseed =3D 279534051 mnt_secondary_writes =3D 0 mnt_secondary_accwrites =3D 0 mnt_gjprovider =3D NULL List of active vnodes vnode vnode 0xfffffe01077375e8: 0xfffffe01077375e8: tag null, type VDIR tag null, type VDIR usecount 2, writecount 0, refcount 2 mountedhere 0 usecount 2, writecount 0, refcount 2 mountedhere 0 flags (VI_ACTIVE) flags (VI_ACTIVE) v_object 0xfffffe006c8d30e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe006c8d30e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: EXCL by thread 0xfffffe0033498920 (pid 1668) lock type zfs: EXCL by thread 0xfffffe0033498920 (pid 1668) vp=3D0xfffffe01077375e8, lowervp=3D0xfffffe001dbc53f0 vp=3D0xfffffe01077375e8, lowervp=3D0xfffffe001dbc53f0 vnode vnode 0xfffffe006cdf79d8: 0xfffffe006cdf79d8: tag null, type VDIR tag null, type VDIR usecount 3, writecount 0, refcount 3 mountedhere 0 usecount 3, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) flags (VI_ACTIVE) v_object 0xfffffe006c76eae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe006c76eae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cdf79d8, lowervp=3D0xfffffe006cdf77e0 vp=3D0xfffffe006cdf79d8, lowervp=3D0xfffffe006cdf77e0 vnode vnode 0xfffffe006cb67000: 0xfffffe006cb67000: tag null, type VDIR tag null, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0xfffffe001dc0e9a8 usecount 1, writecount 0, refcount 1 mountedhere 0xfffffe001dc0e9a8 flags (VI_ACTIVE) flags (VI_ACTIVE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cb67000, lowervp=3D0xfffffe006cb671f8 vp=3D0xfffffe006cb67000, lowervp=3D0xfffffe006cb671f8 vnode vnode 0xfffffe006cb869d8: 0xfffffe006cb869d8: tag syncer, type VNON= tag syncer, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE) flags (VI_ACTIVE) lock type syncer: UNLOCKED lock type syncer: UNLOCKED vnode vnode 0xfffffe006cb86bd0: 0xfffffe006cb86bd0: tag null, type VDIR tag null, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) flags (VV_ROOT|VI_ACTIVE) v_object 0xfffffe006c8d30e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe006c8d30e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: EXCL by thread 0xfffffe0033498920 (pid 1668) lock type zfs: EXCL by thread 0xfffffe0033498920 (pid 1668) vp=3D0xfffffe006cb86bd0, lowervp=3D0xfffffe001dbc53f0 vp=3D0xfffffe006cb86bd0, lowervp=3D0xfffffe001dbc53f0 List of inactive vnodes vnode vnode 0xfffffe006cd8a1f8: 0xfffffe006cd8a1f8: tag null, type VDIR tag null, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe006c718cb0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe006c718cb0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cd8a1f8, lowervp=3D0xfffffe006c895bd0 vp=3D0xfffffe006cd8a1f8, lowervp=3D0xfffffe006c895bd0 vnode vnode 0xfffffe006cebf1f8: 0xfffffe006cebf1f8: tag null, type VDIR tag null, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe006ce740e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe006ce740e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cebf1f8, lowervp=3D0xfffffe006cebf3f0 vp=3D0xfffffe006cebf1f8, lowervp=3D0xfffffe006cebf3f0 vnode vnode 0xfffffe006cdf6000: 0xfffffe006cdf6000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cdf6000, lowervp=3D0xfffffe006cddfdc8 vp=3D0xfffffe006cdf6000, lowervp=3D0xfffffe006cddfdc8 vnode vnode 0xfffffe006cdf63f0: 0xfffffe006cdf63f0: tag null, type VDIR tag null, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe006ce75570 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe006ce75570 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cdf63f0, lowervp=3D0xfffffe006cdf61f8 vp=3D0xfffffe006cdf63f0, lowervp=3D0xfffffe006cdf61f8 vnode vnode 0xfffffe006cdf6dc8: 0xfffffe006cdf6dc8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cdf6dc8, lowervp=3D0xfffffe006cdf65e8 vp=3D0xfffffe006cdf6dc8, lowervp=3D0xfffffe006cdf65e8 vnode vnode 0xfffffe006cdf75e8: 0xfffffe006cdf75e8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cdf75e8, lowervp=3D0xfffffe006cdf73f0 vp=3D0xfffffe006cdf75e8, lowervp=3D0xfffffe006cdf73f0 vnode vnode 0xfffffe006cdf7dc8: 0xfffffe006cdf7dc8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe006cdd7ae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe006cdd7ae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cdf7dc8, lowervp=3D0xfffffe006cdf7bd0 vp=3D0xfffffe006cdf7dc8, lowervp=3D0xfffffe006cdf7bd0 vnode vnode 0xfffffe006cde41f8: 0xfffffe006cde41f8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cde41f8, lowervp=3D0xfffffe006cde4000 vp=3D0xfffffe006cde41f8, lowervp=3D0xfffffe006cde4000 vnode vnode 0xfffffe006cde45e8: 0xfffffe006cde45e8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cde45e8, lowervp=3D0xfffffe006cde43f0 vp=3D0xfffffe006cde45e8, lowervp=3D0xfffffe006cde43f0 vnode vnode 0xfffffe006cde49d8: 0xfffffe006cde49d8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cde49d8, lowervp=3D0xfffffe006cde47e0 vp=3D0xfffffe006cde49d8, lowervp=3D0xfffffe006cde47e0 vnode vnode 0xfffffe0033460bd0: 0xfffffe0033460bd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe0033460bd0, lowervp=3D0xfffffe006c56e000 vp=3D0xfffffe0033460bd0, lowervp=3D0xfffffe006c56e000 vnode vnode 0xfffffe006cfa85e8: 0xfffffe006cfa85e8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cfa85e8, lowervp=3D0xfffffe006cde51f8 vp=3D0xfffffe006cfa85e8, lowervp=3D0xfffffe006cde51f8 vnode vnode 0xfffffe006cfa81f8: 0xfffffe006cfa81f8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cfa81f8, lowervp=3D0xfffffe006cfa83f0 vp=3D0xfffffe006cfa81f8, lowervp=3D0xfffffe006cfa83f0 vnode vnode 0xfffffe006cfa7dc8: 0xfffffe006cfa7dc8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cfa7dc8, lowervp=3D0xfffffe006cfa8000 vp=3D0xfffffe006cfa7dc8, lowervp=3D0xfffffe006cfa8000 vnode vnode 0xfffffe006cfa79d8: 0xfffffe006cfa79d8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cfa79d8, lowervp=3D0xfffffe006cfa7bd0 vp=3D0xfffffe006cfa79d8, lowervp=3D0xfffffe006cfa7bd0 vnode vnode 0xfffffe006cfa75e8: 0xfffffe006cfa75e8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cfa75e8, lowervp=3D0xfffffe006cfa77e0 vp=3D0xfffffe006cfa75e8, lowervp=3D0xfffffe006cfa77e0 vnode vnode 0xfffffe006cfa71f8: 0xfffffe006cfa71f8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cfa71f8, lowervp=3D0xfffffe006cfa73f0 vp=3D0xfffffe006cfa71f8, lowervp=3D0xfffffe006cfa73f0 vnode vnode 0xfffffe006cfa6dc8: 0xfffffe006cfa6dc8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe006cfa6dc8, lowervp=3D0xfffffe006cfa7000 vp=3D0xfffffe006cfa6dc8, lowervp=3D0xfffffe006cfa7000 vnode vnode 0xfffffe01077377e0: 0xfffffe01077377e0: tag null, type VDIR tag null, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe006c718cb0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe006c718cb0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01077377e0, lowervp=3D0xfffffe006c895bd0 vp=3D0xfffffe01077377e0, lowervp=3D0xfffffe006c895bd0 Hope this is more useful. Maybe these LORs are not very interesting (haven't check if they are on the begnin list), but I get these now after raising WITNESS_COUNT: lock order reversal: 1st 0xfffffe0007b3f680 unionfs (unionfs) @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/fs/unionfs/union_subr.c:= 368 2nd 0xfffffe000779e098 ufs (ufs) @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/vfs_subr.c:2337 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff82e10ce180 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e10ce240 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff82e10ce260 witness_checkorder() at witness_checkorder+0x875/frame 0xffffff82e10ce320= __lockmgr_args() at __lockmgr_args+0x1197/frame 0xffffff82e10ce400 ffs_lock() at ffs_lock+0x9c/frame 0xffffff82e10ce450 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe3/frame 0xffffff82e10ce480 _vn_lock() at _vn_lock+0x57/frame 0xffffff82e10ce4e0 vputx() at vputx+0x310/frame 0xffffff82e10ce540 unionfs_noderem() at unionfs_noderem+0x1d7/frame 0xffffff82e10ce5c0 unionfs_reclaim() at unionfs_reclaim+0x11/frame 0xffffff82e10ce5d0 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x105/frame 0xffffff82e10ce600 vgonel() at vgonel+0x12f/frame 0xffffff82e10ce670 vrecycle() at vrecycle+0x5a/frame 0xffffff82e10ce6a0 unionfs_inactive() at unionfs_inactive+0x20/frame 0xffffff82e10ce6b0 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x105/frame 0xffffff82e10ce6e0 vinactive() at vinactive+0xa4/frame 0xffffff82e10ce730 vputx() at vputx+0x425/frame 0xffffff82e10ce790 kern_statat_vnhook() at kern_statat_vnhook+0x15b/frame 0xffffff82e10ce910= kern_statat() at kern_statat+0x15/frame 0xffffff82e10ce930 sys_lstat() at sys_lstat+0x2a/frame 0xffffff82e10ce9d0 amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e10ceaf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e10ceaf0 --- syscall (190, FreeBSD ELF64, sys_lstat), rip =3D 0x80060fa0c, rsp =3D= 0x7fffffffd6d8, rbp =3D 0 --- lock order reversal: 1st 0xffffff826a62da08 bufwait (bufwait) @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/vfs_bio.c:3140 2nd 0xfffffe0033431000 dirhash (dirhash) @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/ufs/ufs/ufs_dirhash.c:28= 4 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff82e114b2d0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e114b390 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff82e114b3b0 witness_checkorder() at witness_checkorder+0x875/frame 0xffffff82e114b470= _sx_xlock() at _sx_xlock+0x70/frame 0xffffff82e114b4a0 ufsdirhash_acquire() at ufsdirhash_acquire+0x44/frame 0xffffff82e114b4c0 ufsdirhash_add() at ufsdirhash_add+0x19/frame 0xffffff82e114b4f0 ufs_direnter() at ufs_direnter+0x6d8/frame 0xffffff82e114b5c0 ufs_mkdir() at ufs_mkdir+0x501/frame 0xffffff82e114b7b0 VOP_MKDIR_APV() at VOP_MKDIR_APV+0x105/frame 0xffffff82e114b7e0 kern_mkdirat() at kern_mkdirat+0x290/frame 0xffffff82e114b9d0 amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e114baf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e114baf0 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip =3D 0x80091324c, rsp =3D= 0x7fffffffe778, rbp =3D 0x800c07050 --- Thanks, -Harry --------------enig38F03A934E680565AC552C34 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlPTgLIACgkQLDqVQ9VXb8g6dQCdHS4v7yicMkVrUQiFAH88lQdS 4zQAn3eHF0HJ/LEWg5jl61/nPUkuYpgt =OAOM -----END PGP SIGNATURE----- --------------enig38F03A934E680565AC552C34--