Date: Mon, 24 Oct 2022 17:41:10 +0200 From: Mateusz Guzik <mjguzik@gmail.com> To: Andriy Gapon <avg@freebsd.org> Cc: freebsd-stable List <stable@freebsd.org>, freebsd-fs <fs@freebsd.org> Subject: Re: panic: condition seqc_in_modify not met, while replaying ZIL Message-ID: <CAGudoHH-_Mq23nzm2NQ8n2R7=pDQdfV%2Bk=AbbUrWH_XGoYTQzQ@mail.gmail.com> In-Reply-To: <2f586acb-a8ee-fa17-bbbb-740f41dbfe82@FreeBSD.org> References: <93085f3c-4592-8f46-fadb-ba0ee2d2ffa9@FreeBSD.org> <CAGudoHH5iq2DmK5z0C8hAD=NF8_PAEQW4s43b%2B-G162xK7efrg@mail.gmail.com> <2f586acb-a8ee-fa17-bbbb-740f41dbfe82@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 10/24/22, Andriy Gapon <avg@freebsd.org> wrote: > On 20/10/2022 17:32, Mateusz Guzik wrote: >> This is a false-positive -- the seqc stuff is not modified on replay >> and zfs internal asserts check for it. I think the easiest way forward >> for this one is to appease the assert by entering it seqc. >> >> That said, could it be it is an invariant there are no nc entries at >> this stage anyway and the call to cache_vop_rmdir can be avoided? I >> can add cache_assert_no_entries or similar routine to be called >> instead. > > I've checked the code and, indeed, the name cache is doubly disabled while > ZIL > is replayed. ZFS sets z_use_namecache to false and z_replay to true before > the > replay. And either is sufficient to disable ZFS name cache calls (which is > > redundant, but that's a different topic). > > So, I came up with this small proof-of-concept patch (and it helped with the > > problem at hand). > It does not assert that the name cache for the filesystem being mounted is > actually empty. That would be a good addition, if you wish to add it. > > diff --git a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > index f6bc9c0c6afb..87fa271dfd5c 100644 > --- a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > +++ b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c > @@ -1622,7 +1622,8 @@ zfs_rmdir_(vnode_t *dvp, vnode_t *vp, const char > *name, > cred_t *cr) > > dmu_tx_commit(tx); > > - cache_vop_rmdir(dvp, vp); > + if (zfsvfs->z_use_namecache) > + cache_vop_rmdir(dvp, vp); > out: > if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS) > zil_commit(zilog, 0); > Added the routine in https://cgit.FreeBSD.org/src/commit/?id=d653aaec7a311a7d1455e9721904b2a341b32a65, just call it for the else clause. > > >> On 10/20/22, Andriy Gapon <avg@freebsd.org> wrote: >>> >>> This happens on stable/13, custom kernel compiled with DEBUG_VFS_LOCKS, >>> from >>> >>> mid-September after an ungraceful reboot (unrelated crash). As far as I >>> can >>> see >>> in kgdb, both dvp and vp have v_seqc == 0. >>> >>> VNASSERT failed: ({ seqc_t __seqc = (_vp->v_seqc); >>> __builtin_expect((__seqc >>> & >>> 1), 0); }) not true at >>> /usr/home/avg/devel/freebsd-src-new/machines/trant/sys/kern/vfs_cache.c:2976 >>> >>> (cache_vop_rmdir) >>> 0xfffff8004bc0bd58: type VDIR >>> usecount 1, writecount 0, refcount 1 seqc users 0 mountedhere 0 >>> hold count flags () >>> flags () >>> lock type zfs: UNLOCKED >>> panic: condition seqc_in_modify(_vp->v_seqc) not met at >>> /usr/home/avg/devel/freebsd-src-new/machines/trant/sys/kern/vfs_cache.c:2976 >>> >>> (cache_vop_rmdir) >>> cpuid = 5 >>> time = 1666250711 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at 0xffffffff8061549b = >>> db_trace_self_wrapper+0x2b/frame >>> 0xfffffe01ce6c7090 >>> kdb_backtrace() at 0xffffffff80942927 = kdb_backtrace+0x37/frame >>> 0xfffffe01ce6c7140 >>> vpanic() at 0xffffffff808f4fb4 = vpanic+0x184/frame 0xfffffe01ce6c71a0 >>> panic() at 0xffffffff808f4d63 = panic+0x43/frame 0xfffffe01ce6c7200 >>> cache_vop_rmdir() at 0xffffffff809b9b9f = cache_vop_rmdir+0xdf/frame >>> 0xfffffe01ce6c7220 >>> zfs_rmdir_() at 0xffffffff80391cff = zfs_rmdir_+0x1df/frame >>> 0xfffffe01ce6c7290 >>> zfs_rmdir() at 0xffffffff80391af8 = zfs_rmdir+0x48/frame >>> 0xfffffe01ce6c7310 >>> zfs_replay_remove() at 0xffffffff804fc53b = zfs_replay_remove+0x7b/frame >>> 0xfffffe01ce6c7340 >>> zil_replay_log_record() at 0xffffffff80508462 = >>> zil_replay_log_record+0x212/frame 0xfffffe01ce6c7490 >>> zil_parse() at 0xffffffff80502f4b = zil_parse+0x5cb/frame >>> 0xfffffe01ce6c76a0 >>> zil_replay() at 0xffffffff805081d8 = zil_replay+0xd8/frame >>> 0xfffffe01ce6c7700 >>> zfsvfs_setup() at 0xffffffff8038f09d = zfsvfs_setup+0x24d/frame >>> 0xfffffe01ce6c7930 >>> zfs_mount() at 0xffffffff8038c9b2 = zfs_mount+0x652/frame >>> 0xfffffe01ce6c7ad0 >>> vfs_domount_first() at 0xffffffff809d0d66 = >>> vfs_domount_first+0x216/frame >>> 0xfffffe01ce6c7c00 >>> vfs_domount() at 0xffffffff809cdd43 = vfs_domount+0x2d3/frame >>> 0xfffffe01ce6c7d30 >>> vfs_donmount() at 0xffffffff809ccb6f = vfs_donmount+0x81f/frame >>> 0xfffffe01ce6c7dc0 >>> sys_nmount() at 0xffffffff809cc318 = sys_nmount+0x108/frame >>> 0xfffffe01ce6c7df0 >>> amd64_syscall() at 0xffffffff80c31d16 = amd64_syscall+0x186/frame >>> 0xfffffe01ce6c7f30 >>> fast_syscall_common() at 0xffffffff80c0889b = >>> fast_syscall_common+0xf8/frame >>> >>> 0xfffffe01ce6c7f30 >>> --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0xb6aac9dd1a, rsp = >>> 0xb6b538f468, rbp = 0xb6b538f4d0 --- >>> Uptime: 37s >>> Dumping 1426 out of 32644 MB: (CTRL-C to abort) >>> ..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>> >>> -- >>> Andriy Gapon >>> >> >> > > -- > Andriy Gapon > > -- Mateusz Guzik <mjguzik gmail.com>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGudoHH-_Mq23nzm2NQ8n2R7=pDQdfV%2Bk=AbbUrWH_XGoYTQzQ>