From nobody Mon Oct 24 15:09:11 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mwz4T6VMBz4fwGF; Mon, 24 Oct 2022 15:09:13 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mwz4T5hcyz3Q7T; Mon, 24 Oct 2022 15:09:13 +0000 (UTC) (envelope-from avg@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1666624153; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SG02+7iOKx1wHG7FydUO9S9HJyJzfp/eUVJlZnvvbbw=; b=MbPHpHOotKtKpue2bApBkQ/rpgS6wCJ9N6yvTB1ARH0Y5hPLjKHt95ZScKFM7/qNMj1Y0Q gZ1A+JYsfN/P44VnCCoGywjES90uAeUjKZ8Y9WVfKGICNPvbHuZWCdeLFP5TaXZ5MKAItL Kuk9p2WB4ZyKFHnup+crqUyopOxMnvrYmZtjudymfPHxhKssX/6Jr2ZmKOaDrRqblGdlGm UM8k62EnpCyPOBcW+oqFyT0nzYf3MrApBw7ansuUuR6wtoT3w9wQR2iHEssDj62pAHlgjY D3hwUOqiG9R3zBj6NjzAPdEJNCZ5Ap+CeTHWDYDzZHR30GzVGiU8cRrnIRpXbA== Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Mwz4T1CfFzrDQ; Mon, 24 Oct 2022 15:09:13 +0000 (UTC) (envelope-from avg@freebsd.org) Message-ID: <2f586acb-a8ee-fa17-bbbb-740f41dbfe82@FreeBSD.org> Date: Mon, 24 Oct 2022 18:09:11 +0300 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.4.0 From: Andriy Gapon Subject: Re: panic: condition seqc_in_modify not met, while replaying ZIL Content-Language: en-US To: Mateusz Guzik Cc: freebsd-stable List , freebsd-fs References: <93085f3c-4592-8f46-fadb-ba0ee2d2ffa9@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1666624153; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SG02+7iOKx1wHG7FydUO9S9HJyJzfp/eUVJlZnvvbbw=; b=F/B34JbW6uK/9u2P7EJpWFfjfNSwikKdxs3SpQ1z7uENcOclcmmvIGgTDpWnEjilUPsa7M g89zKQ4/A/6rytlwDOUH3BucH0mtU813m6x9pUkBwiXm7XH23ufdEoFhgWW7vWCH5cqnJe 9SLYIotF0qhInardlNYwrAPfjKp7mPYnW4oVZ0wLbn8un0AspMA/LCFLglcObIJMgiSrwD r7yWh15sMOvLAs1noovFTLGyjO7nggYvgC/+9wVT1me/BvJgWnUeozBl7OwuEe7zNr4pBh v4qspG2i3NTq1K1YUhPCP++wXu4NgaG5mZqpsPsDSMDKcWNOLeO1Vr1LwXk9Cg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1666624153; a=rsa-sha256; cv=none; b=NGXGzWU0dzEZpUgi1lAkYAtAFrsMPTlXNedqQxAQaSWwYI4FUKf7xROm6lya4bl+03UXBy BDv+ybvXZrRm/9cP1UvculDjtqh3wIcNFHKz4YQsGhKBUJ1ef9/FyEmsDleI+V8LwAmqok 4UhHu+BxxEm7XMT6I4qZM3yPFLr6oZhs9CAxKd/Qmo4EcGaSkGtKXNGM+xEHszxzT7jEju Ub4PGGiB0sj/AF4Ug8Cxpa4gnXWe74++aJT3/TmGwWMCvSlHoGM7CsniWEznWS1RVHzz88 1szsSBlUo1QZhSDUNR71fU9Is7NFgUwYiebZ/HHCDMHPuxa4npmsebB79euieg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N On 20/10/2022 17:32, Mateusz Guzik wrote: > This is a false-positive -- the seqc stuff is not modified on replay > and zfs internal asserts check for it. I think the easiest way forward > for this one is to appease the assert by entering it seqc. > > That said, could it be it is an invariant there are no nc entries at > this stage anyway and the call to cache_vop_rmdir can be avoided? I > can add cache_assert_no_entries or similar routine to be called > instead. I've checked the code and, indeed, the name cache is doubly disabled while ZIL is replayed. ZFS sets z_use_namecache to false and z_replay to true before the replay. And either is sufficient to disable ZFS name cache calls (which is redundant, but that's a different topic). So, I came up with this small proof-of-concept patch (and it helped with the problem at hand). It does not assert that the name cache for the filesystem being mounted is actually empty. That would be a good addition, if you wish to add it. diff --git a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c index f6bc9c0c6afb..87fa271dfd5c 100644 --- a/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c +++ b/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c @@ -1622,7 +1622,8 @@ zfs_rmdir_(vnode_t *dvp, vnode_t *vp, const char *name, cred_t *cr) dmu_tx_commit(tx); - cache_vop_rmdir(dvp, vp); + if (zfsvfs->z_use_namecache) + cache_vop_rmdir(dvp, vp); out: if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS) zil_commit(zilog, 0); > On 10/20/22, Andriy Gapon wrote: >> >> This happens on stable/13, custom kernel compiled with DEBUG_VFS_LOCKS, from >> >> mid-September after an ungraceful reboot (unrelated crash). As far as I can >> see >> in kgdb, both dvp and vp have v_seqc == 0. >> >> VNASSERT failed: ({ seqc_t __seqc = (_vp->v_seqc); __builtin_expect((__seqc >> & >> 1), 0); }) not true at >> /usr/home/avg/devel/freebsd-src-new/machines/trant/sys/kern/vfs_cache.c:2976 >> >> (cache_vop_rmdir) >> 0xfffff8004bc0bd58: type VDIR >> usecount 1, writecount 0, refcount 1 seqc users 0 mountedhere 0 >> hold count flags () >> flags () >> lock type zfs: UNLOCKED >> panic: condition seqc_in_modify(_vp->v_seqc) not met at >> /usr/home/avg/devel/freebsd-src-new/machines/trant/sys/kern/vfs_cache.c:2976 >> >> (cache_vop_rmdir) >> cpuid = 5 >> time = 1666250711 >> KDB: stack backtrace: >> db_trace_self_wrapper() at 0xffffffff8061549b = >> db_trace_self_wrapper+0x2b/frame >> 0xfffffe01ce6c7090 >> kdb_backtrace() at 0xffffffff80942927 = kdb_backtrace+0x37/frame >> 0xfffffe01ce6c7140 >> vpanic() at 0xffffffff808f4fb4 = vpanic+0x184/frame 0xfffffe01ce6c71a0 >> panic() at 0xffffffff808f4d63 = panic+0x43/frame 0xfffffe01ce6c7200 >> cache_vop_rmdir() at 0xffffffff809b9b9f = cache_vop_rmdir+0xdf/frame >> 0xfffffe01ce6c7220 >> zfs_rmdir_() at 0xffffffff80391cff = zfs_rmdir_+0x1df/frame >> 0xfffffe01ce6c7290 >> zfs_rmdir() at 0xffffffff80391af8 = zfs_rmdir+0x48/frame 0xfffffe01ce6c7310 >> zfs_replay_remove() at 0xffffffff804fc53b = zfs_replay_remove+0x7b/frame >> 0xfffffe01ce6c7340 >> zil_replay_log_record() at 0xffffffff80508462 = >> zil_replay_log_record+0x212/frame 0xfffffe01ce6c7490 >> zil_parse() at 0xffffffff80502f4b = zil_parse+0x5cb/frame >> 0xfffffe01ce6c76a0 >> zil_replay() at 0xffffffff805081d8 = zil_replay+0xd8/frame >> 0xfffffe01ce6c7700 >> zfsvfs_setup() at 0xffffffff8038f09d = zfsvfs_setup+0x24d/frame >> 0xfffffe01ce6c7930 >> zfs_mount() at 0xffffffff8038c9b2 = zfs_mount+0x652/frame >> 0xfffffe01ce6c7ad0 >> vfs_domount_first() at 0xffffffff809d0d66 = vfs_domount_first+0x216/frame >> 0xfffffe01ce6c7c00 >> vfs_domount() at 0xffffffff809cdd43 = vfs_domount+0x2d3/frame >> 0xfffffe01ce6c7d30 >> vfs_donmount() at 0xffffffff809ccb6f = vfs_donmount+0x81f/frame >> 0xfffffe01ce6c7dc0 >> sys_nmount() at 0xffffffff809cc318 = sys_nmount+0x108/frame >> 0xfffffe01ce6c7df0 >> amd64_syscall() at 0xffffffff80c31d16 = amd64_syscall+0x186/frame >> 0xfffffe01ce6c7f30 >> fast_syscall_common() at 0xffffffff80c0889b = fast_syscall_common+0xf8/frame >> >> 0xfffffe01ce6c7f30 >> --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0xb6aac9dd1a, rsp = >> 0xb6b538f468, rbp = 0xb6b538f4d0 --- >> Uptime: 37s >> Dumping 1426 out of 32644 MB: (CTRL-C to abort) >> ..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> -- >> Andriy Gapon >> > > -- Andriy Gapon