Date: Tue, 14 Apr 2026 21:42:05 +0200 From: Jan Martin Mikkelsen <janm@transactionware.com> To: Konstantin Belousov <kib@freebsd.org> Cc: current@freebsd.org Subject: Re: Panic: cache_vop_rename: lingering negative entry Message-ID: <019D4A17-B8AF-4768-ADC4-9BDF80BF566B@transactionware.com> In-Reply-To: <8E5A88B4-1EDC-415B-BF35-45AED0B18042@transactionware.com> References: <2016260A-5C07-45EE-87CA-73918BA16E83@transactionware.com> <adU2ibwH_2aeq8m9@kib.kiev.ua> <44E3FE9A-4244-49EB-97E0-16080B68F12B@transactionware.com> <EC385E00-435E-4EF0-BE8E-7D1337E64E80@transactionware.com> <ad1OTEZbzeerf2Jl@kib.kiev.ua> <0610AE32-DD37-401B-BA04-8C092D61C8B3@transactionware.com> <ad4Odea_ItBembsx@kib.kiev.ua> <8E5A88B4-1EDC-415B-BF35-45AED0B18042@transactionware.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
> On 14. Apr 2026, at 17:57, Jan Martin Mikkelsen <janm@transactionware.com> wrote:
>
>
>> On 14 Apr 2026, at 11:52, Konstantin Belousov <kib@freebsd.org> wrote:
>>
>> On Tue, Apr 14, 2026 at 11:45:08AM +0200, Jan Martin Mikkelsen wrote:
>>>
>>>> On 13 Apr 2026, at 22:13, Konstantin Belousov <kib@freebsd.org> wrote:
>>>>
>>>> On Mon, Apr 13, 2026 at 07:12:32PM +0200, Jan Martin Mikkelsen wrote:
>>>>>
>>>>>> On 7 Apr 2026, at 20:20, Jan Martin Mikkelsen <janm@transactionware.com> wrote:
>>>>>>
>>>>>> On 7 Apr 2026, at 18:53, Konstantin Belousov <kib@freebsd.org> wrote:
>>>>>>>
>>>>>>> On Tue, Apr 07, 2026 at 05:02:05PM +0200, Jan Martin Mikkelsen wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am consistently getting the panic below while building lang/perl5.42. This is the command from the perl build that triggers the panic:
>>>>>>>>
>>>>>>>> /usr/bin/strip /ports-work/usr/ports/lang/perl5.42/work/stage/usr/local/bin/perl5.42.0
>>>>>>>>
>>>>>>>> CURRENT on aarch64, with a kernel from last week, also with a later one from the weekend. A kernel from mid-January worked fine.
>>>>>>>>
>>>>>>>> I can reproduce on demand, no parallelism in the build required.
>>>>>>>>
>>>>>>>> Does this look familiar to anyone?
>>>>>>>>
>>>>>>>> panic: cache_vop_rename: lingering negative entry
>>>>>>>> cpuid = 4
>>>>>>>> time = 1775410763
>>>>>>>> KDB: stack backtrace:
>>>>>>>> db_trace_self() at db_trace_self
>>>>>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x38
>>>>>>>> vpanic() at vpanic+0x1a0
>>>>>>>> panic() at panic+0x48
>>>>>>>> cache_vop_rename() at cache_vop_rename+0xb0
>>>>>>>> zfs_do_rename() at zfs_do_rename+0xafc
>>>>>>>> zfs_freebsd_rename() at zfs_freebsd_rename+0x5c
>>>>>>>> VOP_RENAME_APV() at VOP_RENAME_APV+0x44
>>>>>>>> kern_renameat () at kern_renameat+0x574
>>>>>>>> do_el0_sync() at do_el0_sync+0x5f8
>>>>>>>> handle_el0_sync() at handle_el0_sync+0x4c
>>>>>>>> --- exception, esr 0x56000000
>>>>>>>> KDB: enter: panic
>>>>>>>> [ thread pid 81230 tid 101738 ]
>>>>>>>> Stopped at kdb_enter+0x48: str xzr, [x19, #3072]
>>>>>>>
>>>>>>> Is it reproducable on UFS and/or tmpfs?
>>>>>>
>>>>>> Successful completion (no panic) when the work directory is on UFS, and when the work directory is on tmpfs. I didn’t try multiple times, but it never works on ZFS.
>>>>>
>>>>> The panic consistently reproduces on a ZFS filesystem with the properties “utf8only=on” and "normalization=formD”.
>>>>>
>>>>> A ZFS file system with “utf8only=off” and "normalization=none” works fine.
>>>>>
>>>>> As far as I can see, strip makes a simple rename(2) call, and testing rename(2) works fine (as expected). Running the same strip command on the same files on a fresh system works fine.
>>>>>
>>>>> The smallest reproducer I have at the moment is building lang/perl5.42.0 with a workdir on a ZFS filesystem enforcing UTF8.
>>>>
>>>> I am now sure that the reason is that the options you used cause the same
>>>> inode to have more than one name (but not hardlinks). I remember that
>>>> zfs had option to be case-insensitive, but I may mis-remember.
>>>>
>>>> The solution, in any case, is to either stop using namecache when these
>>>> options are activated, or at least purge all cached entries that has the
>>>> given dst when the dst vnode is renamed or deleted.
>>>>
>>>> Somebody who knows zfs would be needed to make the change.
>>>
>>> I had a look at the ZFS source, and found this:
>>>
>>> /*
>>> * Only use the name cache if we are looking for a
>>> * name on a file system that does not require normalization
>>> * or case folding. We can also look there if we happen to be
>>> * on a non-normalizing, mixed sensitivity file system IF we
>>> * are looking for the exact name (which is always the case on
>>> * FreeBSD).
>>> */
>>> zfsvfs->z_use_namecache = !zfsvfs->z_norm ||
>>> ((zfsvfs->z_case == ZFS_CASE_MIXED) &&
>>> !(zfsvfs->z_norm & ~U8_TEXTPREP_TOUPPER));
>>>
>>>
>>> The call to cache_vop_rename() which causes the panic is not protected by an “if (zfsvfs->z_use_namecache)”, unlike the rest of the code that uses that to decide whether or not to use the namecache.
>>>
>>> Elsewhere in zfs_vnops_os.c, there is another call to a cache_vop* function, which is protected by a test:
>>>
>>> if (zfsvfs->z_use_namecache)
>>> cache_vop_rmdir(dvp, vp);
>>>
>>> It seems to me that this patch could resolve the problem. Does this seem reasonable?
>>>
>>> --- a/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c 2026-03-28 20:55:06.000000000 1100
>>> +++ b/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c 2026-03-28 20:55:06.000000000 1100
>>> @@ -3524,7 +3524,7 @@
>>> ZRENAMING, NULL));
>>> }
>>> }
>>> - if (error == 0) {
>>> + if (error == 0 && zfsvfs->z_use_namecache) {
>>> cache_vop_rename(sdvp, *svpp, tdvp, *tvpp, scnp, tcnp);
>>> }
>>> }
>>>
>>
>> Yes, but please test.
>> If works for you, please either create a Github PR or a review on the
>> FreeBSD' phab.
>
> That does seem to fix the panic. I’ll do a GitHub PR. Thanks for your help.
https://github.com/openzfs/zfs/pull/18430
[-- Attachment #2 --]
<html aria-label="message body"><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><br><div><blockquote type="cite"><div>On 14. Apr 2026, at 17:57, Jan Martin Mikkelsen <janm@transactionware.com> wrote:</div><br class="Apple-interchange-newline"><div><div><br><blockquote type="cite">On 14 Apr 2026, at 11:52, Konstantin Belousov <kib@freebsd.org> wrote:<br><br>On Tue, Apr 14, 2026 at 11:45:08AM +0200, Jan Martin Mikkelsen wrote:<br><blockquote type="cite"><br><blockquote type="cite">On 13 Apr 2026, at 22:13, Konstantin Belousov <kib@freebsd.org> wrote:<br><br>On Mon, Apr 13, 2026 at 07:12:32PM +0200, Jan Martin Mikkelsen wrote:<br><blockquote type="cite"><br><blockquote type="cite">On 7 Apr 2026, at 20:20, Jan Martin Mikkelsen <janm@transactionware.com> wrote:<br><br>On 7 Apr 2026, at 18:53, Konstantin Belousov <kib@freebsd.org> wrote:<br><blockquote type="cite"><br>On Tue, Apr 07, 2026 at 05:02:05PM +0200, Jan Martin Mikkelsen wrote:<br><blockquote type="cite">Hi,<br><br>I am consistently getting the panic below while building lang/perl5.42. This is the command from the perl build that triggers the panic:<br><br>/usr/bin/strip /ports-work/usr/ports/lang/perl5.42/work/stage/usr/local/bin/perl5.42.0<br><br>CURRENT on aarch64, with a kernel from last week, also with a later one from the weekend. A kernel from mid-January worked fine.<br><br>I can reproduce on demand, no parallelism in the build required.<br><br>Does this look familiar to anyone?<br><br>panic: cache_vop_rename: lingering negative entry<br>cpuid = 4<br>time = 1775410763<br>KDB: stack backtrace:<br>db_trace_self() at db_trace_self<br>db_trace_self_wrapper() at db_trace_self_wrapper+0x38<br>vpanic() at vpanic+0x1a0<br>panic() at panic+0x48<br>cache_vop_rename() at cache_vop_rename+0xb0<br>zfs_do_rename() at zfs_do_rename+0xafc<br>zfs_freebsd_rename() at zfs_freebsd_rename+0x5c<br>VOP_RENAME_APV() at VOP_RENAME_APV+0x44<br>kern_renameat () at kern_renameat+0x574<br>do_el0_sync() at do_el0_sync+0x5f8<br>handle_el0_sync() at handle_el0_sync+0x4c<br>--- exception, esr 0x56000000<br>KDB: enter: panic<br>[ thread pid 81230 tid 101738 ]<br>Stopped at kdb_enter+0x48: str xzr, [x19, #3072]<br></blockquote><br>Is it reproducable on UFS and/or tmpfs?<br></blockquote><br>Successful completion (no panic) when the work directory is on UFS, and when the work directory is on tmpfs. I didn’t try multiple times, but it never works on ZFS.<br></blockquote><br>The panic consistently reproduces on a ZFS filesystem with the properties “utf8only=on” and "normalization=formD”.<br><br>A ZFS file system with “utf8only=off” and "normalization=none” works fine.<br><br>As far as I can see, strip makes a simple rename(2) call, and testing rename(2) works fine (as expected). Running the same strip command on the same files on a fresh system works fine.<br><br>The smallest reproducer I have at the moment is building lang/perl5.42.0 with a workdir on a ZFS filesystem enforcing UTF8.<br></blockquote><br>I am now sure that the reason is that the options you used cause the same<br>inode to have more than one name (but not hardlinks). I remember that<br>zfs had option to be case-insensitive, but I may mis-remember.<br><br>The solution, in any case, is to either stop using namecache when these<br>options are activated, or at least purge all cached entries that has the<br>given dst when the dst vnode is renamed or deleted.<br><br>Somebody who knows zfs would be needed to make the change.<br></blockquote><br>I had a look at the ZFS source, and found this:<br><br> /*<br> * Only use the name cache if we are looking for a<br> * name on a file system that does not require normalization<br> * or case folding. We can also look there if we happen to be<br> * on a non-normalizing, mixed sensitivity file system IF we<br> * are looking for the exact name (which is always the case on<br> * FreeBSD).<br> */<br> zfsvfs->z_use_namecache = !zfsvfs->z_norm ||<br> ((zfsvfs->z_case == ZFS_CASE_MIXED) &&<br> !(zfsvfs->z_norm & ~U8_TEXTPREP_TOUPPER));<br><br><br>The call to cache_vop_rename() which causes the panic is not protected by an “if (zfsvfs->z_use_namecache)”, unlike the rest of the code that uses that to decide whether or not to use the namecache.<br><br>Elsewhere in zfs_vnops_os.c, there is another call to a cache_vop* function, which is protected by a test:<br><br> if (zfsvfs->z_use_namecache)<br> cache_vop_rmdir(dvp, vp);<br><br>It seems to me that this patch could resolve the problem. Does this seem reasonable?<br><br>--- a/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c<span class="Apple-tab-span" style="white-space:pre"> </span>2026-03-28 20:55:06.000000000 1100<br>+++ b/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c<span class="Apple-tab-span" style="white-space:pre"> </span>2026-03-28 20:55:06.000000000 1100<br>@@ -3524,7 +3524,7 @@<br><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span> ZRENAMING, NULL));<br><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>}<br><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>}<br>-<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>if (error == 0) {<br>+<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>if (error == 0 && zfsvfs->z_use_namecache) {<br><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>cache_vop_rename(sdvp, *svpp, tdvp, *tvpp, scnp, tcnp);<br><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>}<br><span class="Apple-tab-span" style="white-space:pre"> </span>}<br><br></blockquote><br>Yes, but please test.<br>If works for you, please either create a Github PR or a review on the<br>FreeBSD' phab.<br></blockquote><br>That does seem to fix the panic. I’ll do a GitHub PR. Thanks for your help.<br></div></div></blockquote></div><br><div><a href="https://github.com/openzfs/zfs/pull/18430">https://github.com/openzfs/zfs/pull/18430</a></div><div><br></div><div><br></div></body></html>
home |
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?019D4A17-B8AF-4768-ADC4-9BDF80BF566B>