From owner-freebsd-stable@FreeBSD.ORG Sun Jul 27 14:44:54 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0540E1B7 for ; Sun, 27 Jul 2014 14:44:54 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CB6C222F for ; Sun, 27 Jul 2014 14:44:53 +0000 (UTC) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id s6REioSU097753; Sun, 27 Jul 2014 16:44:50 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 36F7D3C7A; Sun, 27 Jul 2014 16:44:49 +0200 (CEST) Message-ID: <53D5105A.6040204@omnilan.de> Date: Sun, 27 Jul 2014 16:44:42 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: panic/lock on 9.3-RELEASE with nullfs/nfs/zfs combination References: <53D12973.3010805@omnilan.de> <20140724165917.GT93733@kib.kiev.ua> <53D1503B.2030200@omnilan.de> <20140724193048.GU93733@kib.kiev.ua> <53D2006C.7090207@omnilan.de> <20140725151448.GY93733@kib.kiev.ua> <53D380B2.2080700@omnilan.de> <20140726114839.GF93733@kib.kiev.ua> In-Reply-To: <20140726114839.GF93733@kib.kiev.ua> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig09804AF5039DD5A9FD7ABF27" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Sun, 27 Jul 2014 16:44:50 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) Cc: FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jul 2014 14:44:54 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig09804AF5039DD5A9FD7ABF27 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Bez=C3=BCglich Konstantin Belousov's Nachricht vom 26.07.2014 13:48 (localtime): > On Sat, Jul 26, 2014 at 12:19:30PM +0200, Harald Schmalzbauer wrote: >> vnode vnode 0xfffffe006cb86bd0: 0xfffffe006cb86bd0: tag null, type VDI= R >> tag null, type VDIR >> usecount 1, writecount 0, refcount 1 mountedhere 0 >> usecount 1, writecount 0, refcount 1 mountedhere 0 >> flags (VV_ROOT|VI_ACTIVE) >> flags (VV_ROOT|VI_ACTIVE) >> v_object 0xfffffe006c8d30e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 >> v_object 0xfffffe006c8d30e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 >> lock type zfs: EXCL by thread 0xfffffe0033498920 (pid 1668) >> lock type zfs: EXCL by thread 0xfffffe0033498920 (pid 1668) >> vp=3D0xfffffe006cb86bd0, lowervp=3D0xfffffe001dbc53f0 >> vp=3D0xfffffe006cb86bd0, lowervp=3D0xfffffe001dbc53f0 >> > It is useful, but still requires more work to get to the issue. From th= e > data you posted, I see that the problem was already reported sometime > this winter. We did not come to any conclusion that time. > > Please, do the following. First, apply this debugging patch and obtain > the same data as you did right now. Hopefully, the assert below would > trigger. > > diff --git a/sys/fs/nullfs/null_vnops.c b/sys/fs/nullfs/null_vnops.c > index 70402e3..bc9772c 100644 > --- a/sys/fs/nullfs/null_vnops.c > +++ b/sys/fs/nullfs/null_vnops.c > @@ -372,6 +372,10 @@ null_lookup(struct vop_lookup_args *ap) > */ > ldvp =3D NULLVPTOLOWERVP(dvp); > vp =3D lvp =3D NULL; > + KASSERT((ldvp->v_vflag & VV_ROOT) =3D=3D 0 || > + ((dvp->v_vflag & VV_ROOT) !=3D 0 && (flags & ISDOTDOT) =3D=3D 0),= > + ("ldvp %p fl %#x dvp %p fl %#x flags %#x", ldvp, ldvp->v_vflag, > + dvp, dvp->v_vflag, flags)); > error =3D VOP_LOOKUP(ldvp, &lvp, cnp); > if (error =3D=3D EJUSTRETURN && (flags & ISLASTCN) && > (dvp->v_mount->mnt_flag & MNT_RDONLY) && I can't emphasize how much I hate my lousy C skills... Got "format '%#x' expects type 'unsigned int', but argument 3 has type 'u_long' [-Wformat]" from the compiler and found out that I can't read/understand any part of your patch; not even the syntax :-( After reading printf(3) I change the 3rd additional line into '("ldvp %p fl %#lx dvp %p fl %#lx flags %#x", ldvp, ldvp->v_vflag,' =E2=80=A6 A blind man may sometimes hit the mark. Here's the console output with an additional (ufs/devfs) LOR prepended: 1st 0xfffffe014bc09098 ufs (ufs) @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/vfs_mount.c:851 2nd 0xfffffe014bd97098 devfs (devfs) @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/vfs_subr.c:2225 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff82e4f7c280 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e4f7c340 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff82e4f7c360 witness_checkorder() at witness_checkorder+0x875/frame 0xffffff82e4f7c420= __lockmgr_args() at __lockmgr_args+0x1197/frame 0xffffff82e4f7c500 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff82e4f7c520 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe3/frame 0xffffff82e4f7c550 _vn_lock() at _vn_lock+0x57/frame 0xffffff82e4f7c5b0 vget() at vget+0x7b/frame 0xffffff82e4f7c600 devfs_allocv() at devfs_allocv+0x13e/frame 0xffffff82e4f7c660 devfs_root() at devfs_root+0x4d/frame 0xffffff82e4f7c6a0 vfs_donmount() at vfs_donmount+0xb89/frame 0xffffff82e4f7c990 sys_nmount() at sys_nmount+0x66/frame 0xffffff82e4f7c9d0 amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e4f7caf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e4f7caf0 --- syscall (378, FreeBSD ELF64, sys_nmount), rip =3D 0x800a94ecc, rsp =3D= 0x7fffffffdc08, rbp =3D 0x7fffffffdc20 --- Here's the panic: panic: ldvp 0xfffffe001bafb000 fl 0x1 dvp 0xfffffe01b4e5a9d8 fl 0 flags 0x120e144 cpuid =3D 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff82e4e88f80 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e4e89040 panic() at panic+0x1cd/frame 0xffffff82e4e89140 null_lookup() at null_lookup+0xa6/frame 0xffffff82e4e891c0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x102/frame 0xffffff82e4e891f0 lookup() at lookup+0x32f/frame 0xffffff82e4e89290 namei() at namei+0x3df/frame 0xffffff82e4e89340 vn_open_cred() at vn_open_cred+0x1e2/frame 0xffffff82e4e894b0 vop_stdvptocnp() at vop_stdvptocnp+0x1af/frame 0xffffff82e4e897e0 null_vptocnp() at null_vptocnp+0xf5/frame 0xffffff82e4e89850 VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x105/frame 0xffffff82e4e89880 vn_vptocnp_locked() at vn_vptocnp_locked+0x15b/frame 0xffffff82e4e89910 vn_fullpath1() at vn_fullpath1+0x100/frame 0xffffff82e4e89970 kern___getcwd() at kern___getcwd+0xd4/frame 0xffffff82e4e899d0 amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e4e89af0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e4e89af0 --- syscall (326, FreeBSD ELF64, sys___getcwd), rip =3D 0x8011a191c, rsp = =3D 0x7fffffffe658, rbp =3D 0x801873400 --- KDB: enter: panic [ thread pid 1676 tid 100824 ] Stopped at kdb_enter+0x3b: movq $0,0x6455e2(%rip) db> show alllocks Process 1676 (vim) thread 0xfffffe0036fb8490 (100824) shared lockmgr zfs (zfs) r =3D 0 (0xfffffe001bafb098) locked @ /usr/local/share/deploy-tools/RELENG_9_3/src/sys/fs/nullfs/null_vnops.c:6= 24 db> show mount =E2=80=A6 0xfffffe0007b03338 /zfs/netshares/deployment/pub/FreeBSD/OmniLAN/ports/ports on /.JAILfbsm/usr/ports (nullfs) =E2=80=A6 db> show mount 0xfffffe0007b03338 0xfffffe0007b03338 /zfs/netshares/deployment/pub/FreeBSD/OmniLAN/ports/ports on /.JAILfbsm/usr/ports (nullfs) mnt_flag =3D LOCAL mnt_kern_flag =3D EXTENDED_SHARED, SHARED_WRITES, LOOKUP_EXCL_DOTDOT, MPSAFE, LOOKUP_SHARED mnt_opt =3D rw, fstype, fspath, target, errmsg mnt_stat =3D { version=3D537068824 type=3D222 flags=3D0x000000001000111c bsize=3D512 iosize=3D131072 blocks=3D6314028651 bfree=3D6309830121 bavail=3D6309830121 files=3D6310125501 ffree=3D6309830121 syncwrites=3D0 asyncwrites=3D0 syncreads=3D0 asyncreads=3D0 namemax=3D255 owner=3D0 fsid=3D[687931140, 41] } mnt_cred =3D { uid=3D0 ruid=3D0 } mnt_ref =3D 56 mnt_gen =3D 1 mnt_nvnodelistsize =3D 56 mnt_activevnodelistsize =3D 5 mnt_writeopcount =3D 0 mnt_maxsymlinklen =3D 0 mnt_iosize_max =3D 65536 mnt_hashseed =3D 308101864 mnt_secondary_writes =3D 0 mnt_secondary_accwrites =3D 0 mnt_gjprovider =3D NULL List of active vnodes vnode vnode 0xfffffe01b4e5a9d8: 0xfffffe01b4e5a9d8: tag null, type VDIR tag null, type VDIR usecount 2, writecount 0, refcount 2 mountedhere 0 usecount 2, writecount 0, refcount 2 mountedhere 0 flags (VI_ACTIVE) flags (VI_ACTIVE) v_object 0xfffffe014bbf4828 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bbf4828 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: SHARED (count 1) lock type zfs: SHARED (count 1) vp=3D0xfffffe01b4e5a9d8, lowervp=3D0xfffffe001bafb000 vp=3D0xfffffe01b4e5a9d8, lowervp=3D0xfffffe001bafb000 vnode vnode 0xfffffe014bf26000: 0xfffffe014bf26000: tag null, type VDIR tag null, type VDIR usecount 3, writecount 0, refcount 3 mountedhere 0 usecount 3, writecount 0, refcount 3 mountedhere 0 flags (VI_ACTIVE) flags (VI_ACTIVE) v_object 0xfffffe01b42ae488 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae488 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bf26000, lowervp=3D0xfffffe014bf261f8 vp=3D0xfffffe014bf26000, lowervp=3D0xfffffe014bf261f8 vnode vnode 0xfffffe014bd633f0: 0xfffffe014bd633f0: tag null, type VDIR tag null, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0xfffffe001bb46000 usecount 1, writecount 0, refcount 1 mountedhere 0xfffffe001bb46000 flags (VI_ACTIVE) flags (VI_ACTIVE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bd633f0, lowervp=3D0xfffffe014bd635e8 vp=3D0xfffffe014bd633f0, lowervp=3D0xfffffe014bd635e8 vnode vnode 0xfffffe014bc08000: 0xfffffe014bc08000: tag syncer, type VNON= tag syncer, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VI_ACTIVE) flags (VI_ACTIVE) lock type syncer: UNLOCKED lock type syncer: UNLOCKED vnode vnode 0xfffffe014bc03dc8: 0xfffffe014bc03dc8: tag null, type VDIR tag null, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 usecount 1, writecount 0, refcount 1 mountedhere 0 flags (VV_ROOT|VI_ACTIVE) flags (VV_ROOT|VI_ACTIVE) v_object 0xfffffe014bbf4828 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bbf4828 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: SHARED (count 1) lock type zfs: SHARED (count 1) vp=3D0xfffffe014bc03dc8, lowervp=3D0xfffffe001bafb000 vp=3D0xfffffe014bc03dc8, lowervp=3D0xfffffe001bafb000 List of inactive vnodes vnode vnode 0xfffffe014bd975e8: 0xfffffe014bd975e8: tag null, type VDIR tag null, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bbf4740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bbf4740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bd975e8, lowervp=3D0xfffffe014b5c43f0 vp=3D0xfffffe014bd975e8, lowervp=3D0xfffffe014b5c43f0 vnode vnode 0xfffffe014bfe1bd0: 0xfffffe014bfe1bd0: tag null, type VDIR tag null, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ca828 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ca828 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bfe1bd0, lowervp=3D0xfffffe014bfe1dc8 vp=3D0xfffffe014bfe1bd0, lowervp=3D0xfffffe014bfe1dc8 vnode vnode 0xfffffe014bf27000: 0xfffffe014bf27000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bf27000, lowervp=3D0xfffffe014bf271f8 vp=3D0xfffffe014bf27000, lowervp=3D0xfffffe014bf271f8 vnode vnode 0xfffffe014bf26bd0: 0xfffffe014bf26bd0: tag null, type VDIR tag null, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014be0a3a0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014be0a3a0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bf26bd0, lowervp=3D0xfffffe014bf26dc8 vp=3D0xfffffe014bf26bd0, lowervp=3D0xfffffe014bf26dc8 vnode vnode 0xfffffe014bf267e0: 0xfffffe014bf267e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bf267e0, lowervp=3D0xfffffe014bf269d8 vp=3D0xfffffe014bf267e0, lowervp=3D0xfffffe014bf269d8 vnode vnode 0xfffffe014bf263f0: 0xfffffe014bf263f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bf263f0, lowervp=3D0xfffffe014bf265e8 vp=3D0xfffffe014bf263f0, lowervp=3D0xfffffe014bf265e8 vnode vnode 0xfffffe014bf5fdc8: 0xfffffe014bf5fdc8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae1d0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae1d0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bf5fdc8, lowervp=3D0xfffffe014bf5fbd0 vp=3D0xfffffe014bf5fdc8, lowervp=3D0xfffffe014bf5fbd0 vnode vnode 0xfffffe014beeddc8: 0xfffffe014beeddc8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf700e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf700e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014beeddc8, lowervp=3D0xfffffe014bd981f8 vp=3D0xfffffe014beeddc8, lowervp=3D0xfffffe014bd981f8 vnode vnode 0xfffffe014be037e0: 0xfffffe014be037e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf9c3a0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf9c3a0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014be037e0, lowervp=3D0xfffffe014bf5f9d8 vp=3D0xfffffe014be037e0, lowervp=3D0xfffffe014bf5f9d8 vnode vnode 0xfffffe014bc0b1f8: 0xfffffe014bc0b1f8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf9b9f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf9b9f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bc0b1f8, lowervp=3D0xfffffe014be8c9d8 vp=3D0xfffffe014bc0b1f8, lowervp=3D0xfffffe014be8c9d8 vnode vnode 0xfffffe014be8c1f8: 0xfffffe014be8c1f8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b4036cb0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b4036cb0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014be8c1f8, lowervp=3D0xfffffe014be8c7e0 vp=3D0xfffffe014be8c1f8, lowervp=3D0xfffffe014be8c7e0 vnode vnode 0xfffffe01b423b3f0: 0xfffffe01b423b3f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf70740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf70740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b423b3f0, lowervp=3D0xfffffe014be8c000 vp=3D0xfffffe01b423b3f0, lowervp=3D0xfffffe014be8c000 vnode vnode 0xfffffe014bfe21f8: 0xfffffe014bfe21f8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bedecb0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bedecb0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014bfe21f8, lowervp=3D0xfffffe014bfe2000 vp=3D0xfffffe014bfe21f8, lowervp=3D0xfffffe014bfe2000 vnode vnode 0xfffffe014b45a5e8: 0xfffffe014b45a5e8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b4036bc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b4036bc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b45a5e8, lowervp=3D0xfffffe0036f2bdc8 vp=3D0xfffffe014b45a5e8, lowervp=3D0xfffffe0036f2bdc8 vnode vnode 0xfffffe00366a0dc8: 0xfffffe00366a0dc8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf15828 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf15828 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe00366a0dc8, lowervp=3D0xfffffe00366147e0 vp=3D0xfffffe00366a0dc8, lowervp=3D0xfffffe00366147e0 vnode vnode 0xfffffe0036614dc8: 0xfffffe0036614dc8: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42441d0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42441d0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe0036614dc8, lowervp=3D0xfffffe0036614bd0 vp=3D0xfffffe0036614dc8, lowervp=3D0xfffffe0036614bd0 vnode vnode 0xfffffe014b602bd0: 0xfffffe014b602bd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bdd39f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bdd39f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b602bd0, lowervp=3D0xfffffe014bfe25e8 vp=3D0xfffffe014b602bd0, lowervp=3D0xfffffe014bfe25e8 vnode vnode 0xfffffe014b6027e0: 0xfffffe014b6027e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf08ae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf08ae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b6027e0, lowervp=3D0xfffffe014b6029d8 vp=3D0xfffffe014b6027e0, lowervp=3D0xfffffe014b6029d8 vnode vnode 0xfffffe014b6023f0: 0xfffffe014b6023f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf6f740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf6f740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b6023f0, lowervp=3D0xfffffe014b6025e8 vp=3D0xfffffe014b6023f0, lowervp=3D0xfffffe014b6025e8 vnode vnode 0xfffffe014b602000: 0xfffffe014b602000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014befc9f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014befc9f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b602000, lowervp=3D0xfffffe014b6021f8 vp=3D0xfffffe014b602000, lowervp=3D0xfffffe014b6021f8 vnode vnode 0xfffffe01b42b3bd0: 0xfffffe01b42b3bd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b4244740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b4244740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b42b3bd0, lowervp=3D0xfffffe01b42b3dc8 vp=3D0xfffffe01b42b3bd0, lowervp=3D0xfffffe01b42b3dc8 vnode vnode 0xfffffe01b42b37e0: 0xfffffe01b42b37e0: tag null, type VREG tag null, type VREG [279/1886] usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf15740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf15740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b42b37e0, lowervp=3D0xfffffe01b42b39d8 vp=3D0xfffffe01b42b37e0, lowervp=3D0xfffffe01b42b39d8 vnode vnode 0xfffffe01b42b33f0: 0xfffffe01b42b33f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bd21910 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bd21910 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b42b33f0, lowervp=3D0xfffffe01b42b35e8 vp=3D0xfffffe01b42b33f0, lowervp=3D0xfffffe01b42b35e8 vnode vnode 0xfffffe01b42b3000: 0xfffffe01b42b3000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf9dae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf9dae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b42b3000, lowervp=3D0xfffffe01b42b31f8 vp=3D0xfffffe01b42b3000, lowervp=3D0xfffffe01b42b31f8 vnode vnode 0xfffffe01b428bbd0: 0xfffffe01b428bbd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae658 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae658 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b428bbd0, lowervp=3D0xfffffe01b423cdc8 vp=3D0xfffffe01b428bbd0, lowervp=3D0xfffffe01b423cdc8 vnode vnode 0xfffffe01b428b7e0: 0xfffffe01b428b7e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf09ae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf09ae0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b428b7e0, lowervp=3D0xfffffe01b428b9d8 vp=3D0xfffffe01b428b7e0, lowervp=3D0xfffffe01b428b9d8 vnode vnode 0xfffffe01b428b3f0: 0xfffffe01b428b3f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bef72b8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bef72b8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b428b3f0, lowervp=3D0xfffffe01b428b5e8 vp=3D0xfffffe01b428b3f0, lowervp=3D0xfffffe01b428b5e8 vnode vnode 0xfffffe01b428b000: 0xfffffe01b428b000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b4246000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b4246000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b428b000, lowervp=3D0xfffffe01b428b1f8 vp=3D0xfffffe01b428b000, lowervp=3D0xfffffe01b428b1f8 vnode vnode 0xfffffe01b428abd0: 0xfffffe01b428abd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b4244488 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b4244488 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b428abd0, lowervp=3D0xfffffe01b428adc8 vp=3D0xfffffe01b428abd0, lowervp=3D0xfffffe01b428adc8 vnode vnode 0xfffffe01b428a7e0: 0xfffffe01b428a7e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae910 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae910 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b428a7e0, lowervp=3D0xfffffe01b428a9d8 vp=3D0xfffffe01b428a7e0, lowervp=3D0xfffffe01b428a9d8 vnode vnode 0xfffffe01b428a3f0: 0xfffffe01b428a3f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe0007a31000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe0007a31000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b428a3f0, lowervp=3D0xfffffe01b428a5e8 vp=3D0xfffffe01b428a3f0, lowervp=3D0xfffffe01b428a5e8 vnode vnode 0xfffffe01b428a000: 0xfffffe01b428a000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b4244570 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b4244570 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b428a000, lowervp=3D0xfffffe01b428a1f8 vp=3D0xfffffe01b428a000, lowervp=3D0xfffffe01b428a1f8 vnode vnode 0xfffffe01b427fbd0: 0xfffffe01b427fbd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe0007a30bc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe0007a30bc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b427fbd0, lowervp=3D0xfffffe014b602dc8 vp=3D0xfffffe01b427fbd0, lowervp=3D0xfffffe014b602dc8 vnode vnode 0xfffffe01b427f7e0: 0xfffffe01b427f7e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42472b8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42472b8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b427f7e0, lowervp=3D0xfffffe01b427f9d8 vp=3D0xfffffe01b427f7e0, lowervp=3D0xfffffe01b427f9d8 vnode vnode 0xfffffe01b427f3f0: 0xfffffe01b427f3f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae0e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae0e8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b427f3f0, lowervp=3D0xfffffe01b427f5e8 vp=3D0xfffffe01b427f3f0, lowervp=3D0xfffffe01b427f5e8 vnode vnode 0xfffffe01b427f000: 0xfffffe01b427f000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42473a0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42473a0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b427f000, lowervp=3D0xfffffe01b427f1f8 vp=3D0xfffffe01b427f000, lowervp=3D0xfffffe01b427f1f8 vnode vnode 0xfffffe01b427ebd0: 0xfffffe01b427ebd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf9cd98 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf9cd98 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b427ebd0, lowervp=3D0xfffffe01b427edc8 vp=3D0xfffffe01b427ebd0, lowervp=3D0xfffffe01b427edc8 vnode vnode 0xfffffe01b427e7e0: 0xfffffe01b427e7e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b427e7e0, lowervp=3D0xfffffe01b427e9d8 vp=3D0xfffffe01b427e7e0, lowervp=3D0xfffffe01b427e9d8 vnode vnode 0xfffffe01b427e3f0: 0xfffffe01b427e3f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae570 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae570 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b427e3f0, lowervp=3D0xfffffe01b427e5e8 vp=3D0xfffffe01b427e3f0, lowervp=3D0xfffffe01b427e5e8 vnode vnode 0xfffffe01b427e000: 0xfffffe01b427e000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae9f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae9f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b427e000, lowervp=3D0xfffffe01b427e1f8 vp=3D0xfffffe01b427e000, lowervp=3D0xfffffe01b427e1f8 vnode vnode 0xfffffe014b5cbbd0: 0xfffffe014b5cbbd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014be0a570 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014be0a570 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b5cbbd0, lowervp=3D0xfffffe01b428bdc8 vp=3D0xfffffe014b5cbbd0, lowervp=3D0xfffffe01b428bdc8 vnode vnode 0xfffffe014b5cb7e0: 0xfffffe014b5cb7e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bf6f910 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bf6f910 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b5cb7e0, lowervp=3D0xfffffe014b5cb9d8 vp=3D0xfffffe014b5cb7e0, lowervp=3D0xfffffe014b5cb9d8 vnode vnode 0xfffffe014b5cb3f0: 0xfffffe014b5cb3f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bedebc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bedebc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b5cb3f0, lowervp=3D0xfffffe014b5cb5e8 vp=3D0xfffffe014b5cb3f0, lowervp=3D0xfffffe014b5cb5e8 vnode vnode 0xfffffe014b5cb000: 0xfffffe014b5cb000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42aebc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42aebc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b5cb000, lowervp=3D0xfffffe014b5cb1f8 vp=3D0xfffffe014b5cb000, lowervp=3D0xfffffe014b5cb1f8 vnode vnode 0xfffffe014b5cabd0: 0xfffffe014b5cabd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae3a0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae3a0 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b5cabd0, lowervp=3D0xfffffe014b5cadc8 vp=3D0xfffffe014b5cabd0, lowervp=3D0xfffffe014b5cadc8 vnode vnode 0xfffffe014b5ca7e0: 0xfffffe014b5ca7e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe01b42ae2b8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe01b42ae2b8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b5ca7e0, lowervp=3D0xfffffe014b5ca9d8 vp=3D0xfffffe014b5ca7e0, lowervp=3D0xfffffe014b5ca9d8 vnode vnode 0xfffffe014b5ca3f0: 0xfffffe014b5ca3f0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bede9f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bede9f8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b5ca3f0, lowervp=3D0xfffffe014b5ca5e8 vp=3D0xfffffe014b5ca3f0, lowervp=3D0xfffffe014b5ca5e8 vnode vnode 0xfffffe014b5ca000: 0xfffffe014b5ca000: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bede000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bede000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe014b5ca000, lowervp=3D0xfffffe014b5ca1f8 vp=3D0xfffffe014b5ca000, lowervp=3D0xfffffe014b5ca1f8 vnode vnode 0xfffffe01b42b4bd0: 0xfffffe01b42b4bd0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bef7000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bef7000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b42b4bd0, lowervp=3D0xfffffe01b427fdc8 vp=3D0xfffffe01b42b4bd0, lowervp=3D0xfffffe01b427fdc8 vnode vnode 0xfffffe01b42b47e0: 0xfffffe01b42b47e0: tag null, type VREG tag null, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014befdbc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014befdbc8 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b42b47e0, lowervp=3D0xfffffe01b42b49d8 vp=3D0xfffffe01b42b47e0, lowervp=3D0xfffffe01b42b49d8 vnode vnode 0xfffffe01b4e5abd0: 0xfffffe01b4e5abd0: tag null, type VDIR tag null, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) flags (VI_FREE) v_object 0xfffffe014bbf4740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 v_object 0xfffffe014bbf4740 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type zfs: UNLOCKED lock type zfs: UNLOCKED vp=3D0xfffffe01b4e5abd0, lowervp=3D0xfffffe014b5c43f0 vp=3D0xfffffe01b4e5abd0, lowervp=3D0xfffffe014b5c43f0 db> > After that, you could try the patch which I posted at winter, but which= > was not tested. I do not know whether it is of any help, and I do need > the debugging information with the patch above before I can make any > conclusions. > > diff --git a/sys/fs/nullfs/null_subr.c b/sys/fs/nullfs/null_subr.c > index fa6c4af..3f74579 100644 > --- a/sys/fs/nullfs/null_subr.c > +++ b/sys/fs/nullfs/null_subr.c > @@ -251,6 +251,7 @@ null_nodeget(mp, lowervp, vpp) > vp->v_type =3D lowervp->v_type; > vp->v_data =3D xp; > vp->v_vnlock =3D lowervp->v_vnlock; > + vp->v_vflag =3D lowervp->v_vflag & VV_ROOT; > error =3D insmntque1(vp, mp, null_insmntque_dtr, xp); > if (error !=3D 0) > return (error); I tested this without the ldvp-debug patch and besides the same ufs/devds LOR nothing else happens with my usual simple end limited test.= Thanks so far, let me know if you want more debug output/testings! -Harry --------------enig09804AF5039DD5A9FD7ABF27 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlPVEGAACgkQLDqVQ9VXb8i6WgCfSw7IXgJVxJTdHSMEPhLwVz+V /ToAoMLFCL+I2xG/b18y3qGWvXd9CJ4E =v3mV -----END PGP SIGNATURE----- --------------enig09804AF5039DD5A9FD7ABF27--