Date: Mon, 24 Oct 2022 18:09:11 +0300 From: Andriy Gapon <avg@freebsd.org> To: Mateusz Guzik <mjguzik@gmail.com> Cc: freebsd-stable List <stable@freebsd.org>, freebsd-fs <fs@freebsd.org> Subject: Re: panic: condition seqc_in_modify not met, while replaying ZIL Message-ID: <2f586acb-a8ee-fa17-bbbb-740f41dbfe82@FreeBSD.org> In-Reply-To: <CAGudoHH5iq2DmK5z0C8hAD=NF8_PAEQW4s43b%2B-G162xK7efrg@mail.gmail.com> References: <93085f3c-4592-8f46-fadb-ba0ee2d2ffa9@FreeBSD.org> <CAGudoHH5iq2DmK5z0C8hAD=NF8_PAEQW4s43b%2B-G162xK7efrg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 20/10/2022 17:32, Mateusz Guzik wrote:
> This is a false-positive -- the seqc stuff is not modified on replay
> and zfs internal asserts check for it. I think the easiest way forward
> for this one is to appease the assert by entering it seqc.
>
> That said, could it be it is an invariant there are no nc entries at
> this stage anyway and the call to cache_vop_rmdir can be avoided? I
> can add cache_assert_no_entries or similar routine to be called
> instead.
I've checked the code and, indeed, the name cache is doubly disabled while ZIL
is replayed. ZFS sets z_use_namecache to false and z_replay to true before the
replay. And either is sufficient to disable ZFS name cache calls (which is
redundant, but that's a different topic).
So, I came up with this small proof-of-concept patch (and it helped with the
problem at hand).
It does not assert that the name cache for the filesystem being mounted is
actually empty. That would be a good addition, if you wish to add it.
diff --git a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c
b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c
index f6bc9c0c6afb..87fa271dfd5c 100644
--- a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c
+++ b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c
@@ -1622,7 +1622,8 @@ zfs_rmdir_(vnode_t *dvp, vnode_t *vp, const char *name,
cred_t *cr)
dmu_tx_commit(tx);
- cache_vop_rmdir(dvp, vp);
+ if (zfsvfs->z_use_namecache)
+ cache_vop_rmdir(dvp, vp);
out:
if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
zil_commit(zilog, 0);
> On 10/20/22, Andriy Gapon <avg@freebsd.org> wrote:
>>
>> This happens on stable/13, custom kernel compiled with DEBUG_VFS_LOCKS, from
>>
>> mid-September after an ungraceful reboot (unrelated crash). As far as I can
>> see
>> in kgdb, both dvp and vp have v_seqc == 0.
>>
>> VNASSERT failed: ({ seqc_t __seqc = (_vp->v_seqc); __builtin_expect((__seqc
>> &
>> 1), 0); }) not true at
>> /usr/home/avg/devel/freebsd-src-new/machines/trant/sys/kern/vfs_cache.c:2976
>>
>> (cache_vop_rmdir)
>> 0xfffff8004bc0bd58: type VDIR
>> usecount 1, writecount 0, refcount 1 seqc users 0 mountedhere 0
>> hold count flags ()
>> flags ()
>> lock type zfs: UNLOCKED
>> panic: condition seqc_in_modify(_vp->v_seqc) not met at
>> /usr/home/avg/devel/freebsd-src-new/machines/trant/sys/kern/vfs_cache.c:2976
>>
>> (cache_vop_rmdir)
>> cpuid = 5
>> time = 1666250711
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at 0xffffffff8061549b =
>> db_trace_self_wrapper+0x2b/frame
>> 0xfffffe01ce6c7090
>> kdb_backtrace() at 0xffffffff80942927 = kdb_backtrace+0x37/frame
>> 0xfffffe01ce6c7140
>> vpanic() at 0xffffffff808f4fb4 = vpanic+0x184/frame 0xfffffe01ce6c71a0
>> panic() at 0xffffffff808f4d63 = panic+0x43/frame 0xfffffe01ce6c7200
>> cache_vop_rmdir() at 0xffffffff809b9b9f = cache_vop_rmdir+0xdf/frame
>> 0xfffffe01ce6c7220
>> zfs_rmdir_() at 0xffffffff80391cff = zfs_rmdir_+0x1df/frame
>> 0xfffffe01ce6c7290
>> zfs_rmdir() at 0xffffffff80391af8 = zfs_rmdir+0x48/frame 0xfffffe01ce6c7310
>> zfs_replay_remove() at 0xffffffff804fc53b = zfs_replay_remove+0x7b/frame
>> 0xfffffe01ce6c7340
>> zil_replay_log_record() at 0xffffffff80508462 =
>> zil_replay_log_record+0x212/frame 0xfffffe01ce6c7490
>> zil_parse() at 0xffffffff80502f4b = zil_parse+0x5cb/frame
>> 0xfffffe01ce6c76a0
>> zil_replay() at 0xffffffff805081d8 = zil_replay+0xd8/frame
>> 0xfffffe01ce6c7700
>> zfsvfs_setup() at 0xffffffff8038f09d = zfsvfs_setup+0x24d/frame
>> 0xfffffe01ce6c7930
>> zfs_mount() at 0xffffffff8038c9b2 = zfs_mount+0x652/frame
>> 0xfffffe01ce6c7ad0
>> vfs_domount_first() at 0xffffffff809d0d66 = vfs_domount_first+0x216/frame
>> 0xfffffe01ce6c7c00
>> vfs_domount() at 0xffffffff809cdd43 = vfs_domount+0x2d3/frame
>> 0xfffffe01ce6c7d30
>> vfs_donmount() at 0xffffffff809ccb6f = vfs_donmount+0x81f/frame
>> 0xfffffe01ce6c7dc0
>> sys_nmount() at 0xffffffff809cc318 = sys_nmount+0x108/frame
>> 0xfffffe01ce6c7df0
>> amd64_syscall() at 0xffffffff80c31d16 = amd64_syscall+0x186/frame
>> 0xfffffe01ce6c7f30
>> fast_syscall_common() at 0xffffffff80c0889b = fast_syscall_common+0xf8/frame
>>
>> 0xfffffe01ce6c7f30
>> --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0xb6aac9dd1a, rsp =
>> 0xb6b538f468, rbp = 0xb6b538f4d0 ---
>> Uptime: 37s
>> Dumping 1426 out of 32644 MB: (CTRL-C to abort)
>> ..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>
>> --
>> Andriy Gapon
>>
>
>
--
Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2f586acb-a8ee-fa17-bbbb-740f41dbfe82>