Date: Thu, 25 Jun 2026 09:10:02 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 296269] zfs panic: NULL deref in zfs_acl_release_nodes during znode reclaim (15.1-RELEASE) Message-ID: <bug-296269-227@https.bugs.freebsd.org/bugzilla/>
index | next in thread | raw e-mail
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296269 Bug ID: 296269 Summary: zfs panic: NULL deref in zfs_acl_release_nodes during znode reclaim (15.1-RELEASE) Product: Base System Version: 15.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: fax@nohik.ee Created attachment 272118 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=272118&action=edit full symbolized "bt full" + registers After upgrading from 14.3-RELEASE to 15.1-RELEASE (a "zpool upgrade" was done as part of the upgrade), the system suffers recurring kernel panics: "page fault while in kernel mode", a supervisor WRITE to virtual address 0x8 in the ZFS ACL teardown path, while a znode is being reclaimed. Crash-dump inspection shows the affected znode's in-memory cached ACL (aclnode list) is internally inconsistent, while the on-disk object is fully intact (verified with zdb). This points to an in-memory defect, not on-disk corruption. Environment ----------- - FreeBSD 15.1-RELEASE releng/15.1-n283562-96841ea08dcf GENERIC amd64 (non-debug kernel; some locals show <optimized out>) - Upgraded from 14.3-RELEASE; zpool upgrade performed during the upgrade - Default NFSv4 ACLs: # zfs get acltype,aclmode,aclinherit tank/<dataset> acltype nfsv4 default aclmode discard default aclinherit restricted default (live zfsvfs in the dump confirms z_acl_type=2, z_acl_mode=0, z_acl_inherit=4) - Affected object carries the trivial three-entry ACL (owner@/group@/everyone@; z_acl_count=3, z_acl_bytes=24). No NFS exports or services involved. - Vnode state at crash time: kern.maxvnodes=8388608, vfs.numvnodes=8388608 (vnode cache at its limit) Panic signature (Host B, python3.11) ------------------------------------- Fatal trap 12: page fault while in kernel mode fault virtual address = 0x8 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff82682d1f current process = 77899 (python3.11) panic: page fault Key registers: rax=0xfffff84432c87000 (head node), rbx=0xfffff83234800300 (aclp), rdi=r14=0xfffff83234800330 (&aclp->z_acl), fault va=0x8. Full register dumps for both hosts are in the attached backtrace. Backtrace (key frames; full "bt full" attached) ------------------------------------------------ #8 list_remove_head (list=0xfffff83234800330) spl/list.c:138 [head=0xfffff84432c87000] #9 zfs_acl_release_nodes (aclp=0xfffff83234800300) zfs/zfs_acl.c:499 #10 zfs_acl_free (aclp=0xfffff83234800300) zfs/zfs_acl.c:508 #11 zfs_znode_free (zp=0xfffffe0aaab93728) zfs/zfs_znode_os.c:1321 #12 zfs_zinactive -> #13 zfs_freebsd_reclaim zfs/zfs_znode_os.c:1297 / zfs_vnops_os.c:5738 #14 VOP_RECLAIM_APV -> vgonel -> vtryrecycle -> vnlru_free_impl -> vn_alloc_hard -> getnewvnode_reserve #22 zfs_zget zfs/zfs_znode_os.c:959 #23 zfs_dirent_lookup -> zfs_dirlook -> zfs_lookup -> zfs_freebsd_lookup -> vfs_cache_lookup -> namei #32 kern_statat (fstatat(2)) What the crash dump shows ------------------------- The node passed to list_remove_node() is a valid, linked node (head = 0xfffff84432c87000) whose forward pointer is NULL: head: { list_next = 0x0, list_prev = 0xfffff83234800338 (= &z_acl.list_head) } list_remove_node() runs: node->list_prev->list_next = node->list_next; // (A) sets // list_head.list_next = NULL node->list_next->list_prev = node->list_prev; // (B) NULL->list_prev // => write to va 0x8 Line (B) dereferences the NULL list_next and writes at offset 0x8 (list_prev), which is exactly the faulting address. (The list_head.list_next = 0x0 visible in the post-mortem dump of z_acl is a side effect of line (A) during the crash, not the pre-existing state.) The list is internally inconsistent: the forward chain (list_head.list_next -> head, head->list_next = NULL) and the back pointer (list_head.list_prev = 0xfffff832e5483bc0) disagree, and z_curr_node (0xfffff832e5483bc0) points to a node already in the post-unlink {list_next=0, list_prev=0} state while it is still referenced and z_acl_count still reports 3. In other words, the cached aclnode list appears to have been (partially) torn down already. Full kgdb structure dumps are attached. On-disk object is intact ------------------------ zdb -dddd tank/<dataset> 84644447 reads cleanly: valid metadata, 100% full, no errors, ACL stored as SA in the 168-byte bonus buffer; pflags 40800000004 match the in-core znode z_pflags. gen 2718637 means the object (and its ACL) predates the 14.3 -> 15.1 upgrade. See attachment. Reproducibility --------------- Process-independent and reproduces across hosts: captured on two independent machines under unrelated processes (du and python3.11), each time with the identical fault signature (write to va 0x8; module-relative offset zfs_acl_free+0x45; the two hosts differ only by zfs.ko load base). It is reached via the standard vnode-reclaim path: with the vnode cache at its limit (vfs.numvnodes == kern.maxvnodes), a lookup drives getnewvnode_reserve() -> vnlru -> vgonel -> VOP_RECLAIM -> zfs_freebsd_reclaim -> ... -> zfs_acl_release_nodes -> list_remove_head. Hypothesis (for maintainers to confirm) --------------------------------------- The on-disk object is clean and predates the upgrade, so this looks like an in-memory defect - most plausibly a double-free / use-after-free of the cached aclnode list, leaving a linked node with a NULL forward pointer that the reclaim-time teardown then walks into. A second, independent anomaly in the same znode is consistent with memory corruption rather than a pure list-logic bug: z_zfsvfs = 0xfffff80157aec001, i.e. the live (intact) zfsvfs_t at 0xfffff80157aec000 with bit 0 set. Since the ACL was written by 14.3-era OpenZFS and the crash only appears after the 14.3 -> 15.1 upgrade, the 15.1 code path that decodes/caches/frees a 14.3-written ACL seems a reasonable starting point. Offered as a lead, not a conclusion. Notes ----- - Non-debug GENERIC kernel, so the list/teardown ASSERTs are compiled out (which is why it faults rather than asserting). - vmcore / core.txt can't be shared publicly due to size and privacy, but I can provide them privately on request. - Dataset name anonymized as <dataset>; the file name 'beiruti' is itself already a pseudonym (length preserved). -- You are receiving this mail because: You are the assignee for the bug.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-296269-227>