From nobody Mon Feb 23 12:06:00 2026 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fKKLw5Vcmz6TFSJ for ; Mon, 23 Feb 2026 12:06:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fKKLw3hBzz4KH7 for ; Mon, 23 Feb 2026 12:06:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1771848360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=bypSaZCndlIcC/54YvQANvl7AiiuCJRzEKfXYwzUYUI=; b=O5ifvmGbuPAzJSpgzv7376BNnXFCi0GhirM54IMOjV2xG81mRQox7LARZbvEYn7+TnEi/h YM0boKWAD0MHt4MaR2NVxw3VmnNcWTOTAPp+c0JdwhY2FNK45r0jQGBCxazrsXvUCyX0JR NUjx9uLBYY7VzIbWGDmldyOtSEsQeD5jOKaj7TaKroPr9X4WjPwE/kH1+ZhcAL7Ob/jWam 7zV2PzTu2n7qFfr0iHZnMUKyxx+WNHzfgRO7I79zdd+WSh/NLiZ+i2CaSm09+hIJETJpea YgVxkq9VRitbRSHNouUhpve8IB6RIA3usmW20vUb3C62LWZgI2uDJ1v2NEMUmw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1771848360; a=rsa-sha256; cv=none; b=NPCQAAoOvOlR1WK0XNktv9rmNm+3ioHgWO5LZ6NWSm13ZzmAcHR2196QDadG4SNDX1Xgm5 /sHt7DiSA9DwKtL6r9+0RaJ3E4jIrKUZS1w/UQsey2boMwOCyhLDE8UGAjLd3dYlpzDQ3W AxrVLk1gQRLMWXPjpLQgGmEE/sGgrK8BDofDGsMQnp3GXELs1ogRlEHfr29p+iePAIg6KR 8wdL7/d3AcGY3Y3at8XcQkHieLeZjUOEh2avdwa9FNzkt5FqK6Bh58cYcpCyh/kO5326eU tR30U08z6rxDiGk8lvlDkkR6ikmcPKOzesUAulAu9XiZvdfZGSNwJxEdqt54nA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1771848360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=bypSaZCndlIcC/54YvQANvl7AiiuCJRzEKfXYwzUYUI=; b=wjVs7mIEYJ2wYrvb4bAUOyTQROGjncDPvuf8lcpevAFgPODx9i1BcOz5pV7MgV41F8Gr5B 8qAQEU2Zysg4ssM070oAt89c20BVK7Szk9rgurAmKDH8ksdUs8c5H7sQNdNq5DILf5uEPe RxNXsiJrXb4X/4qdxC39EMGcmaP9nH7LQb2kqmHgSLh50Qjq3CLgRslhoH0jYT2/Ym7nnu 5WhKCQBAiYTvpnU0m4Qd8huYrWOoEw7/o78YcdEMhyZH5qVZfef6igOh0v59G82fXcbLya cEA5Vw2Ij54yYOp8gz7Gl96lkRXzmxBbYowNUINdFcVFMHJBS4RYVHPmCKQRQQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4fKKLw3H5pzCnK for ; Mon, 23 Feb 2026 12:06:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 61NC60iI021650 for ; Mon, 23 Feb 2026 12:06:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 61NC60vS021649 for bugs@FreeBSD.org; Mon, 23 Feb 2026 12:06:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 293382] Dead lock and kernel crash around closefp_impl Date: Mon, 23 Feb 2026 12:06:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: devgs@ukr.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D293382 Bug ID: 293382 Summary: Dead lock and kernel crash around closefp_impl Product: Base System Version: 14.3-STABLE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: devgs@ukr.net Hi! We've been using 14.4-STABLE for some time now and today a weird issue has popped up. All of the sudden, our multi-threaded network app has deadlocked= on some threads, but not on others. We weren't able to neither attach to it wi= th GDB nor kill it with -9. Hard lock inside the kernel. We've managed to coll= ect a few samples of kernel backtrace for this process with `procstat -kk`. All, basically, identical: PID TID COMM TDNAME KSTACK 91545 101569 - mi_switch+0xbd _sx_xlock_hard+0x4ef kern_close+0x179 amd64_syscall+0x117 fast_syscall_common+0xf8 91545 102281 - mi_switch+0xbd _sx_xlock_hard+0x4ef kern_close+0x179 amd64_syscall+0x117 fast_syscall_common+0xf8 91545 102282 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_timedwait_sig+0x12 _sleep+0x1c1 umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 91545 102283 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_timedwait_sig+0x12 _sleep+0x1c1 kqueue_scan+0xa11 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xdf sys_kevent+0x61 amd64_syscall+0x117 fast_syscall_common+0xf8 91545 102284 mi_switch+0xbd _sleep+0x1f3 knote_fdclose+0xac closefp_impl+0xd0 amd64_syscall+0x117 fast_syscall_common+0xf8 91545 102285 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_timedwait_sig+0x12 _sleep+0x1c1 kqueue_scan+0xa11 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xdf sys_kevent+0x61 amd64_syscall+0x117 fast_syscall_common+0xf8 91545 102286 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_timedwait_sig+0x12 _sleep+0x1c1 kqueue_scan+0xa11 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xdf sys_kevent+0x61 amd64_syscall+0x117 fast_syscall_common+0xf8 Apparently, three threads were deadlocked: first two, that are unnamed and ``. The last one is the thread that is handling inbound socket connections. Hundreds of thousands of them, mostly WebSocket. Two other thr= eads also use sockets, but for outbound connections. During normal operation, sockets are being open and closed as needed, obviously. Seems like in some = case this may lead to a deadlock. Where one thread enters some state in kernel w= here it hangs, holding the lock and preventing others from closing (or modifying descriptors generally). App is async and uses kqueue for networking sockets extensively. We suspect `` to be the culprit, specifically its backtrace where `closefp_impl` is involved. And here's why. When this happened and the traffic was switched to a redund= ancy server, it almost immediately panicked and wend into reboot. Hopefully, we'= ve got the core dump and were able to analyze it somewhat. And there we saw `closefp_impl` from within the same (not physically, different server) thre= ad ``: Fatal trap 12: page fault while in kernel mode cpuid =3D 22; apic id =3D 52 fault virtual address =3D 0x10 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80572e28 stack pointer =3D 0x28:0xfffffe071c126d70 frame pointer =3D 0x28:0xfffffe071c126dc0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 58518 () rdi: fffff83402622be0 rsi: 0000000000000000 rdx: 0000000000000000 rcx: 0000000000000000 r8: fffff80160b9c520 r9: fffffe071c127000 rax: 0000000000000000 rbx: 0000000000031361 rbp: fffffe071c126dc0 r10: 0000000000000001 r11: 0000000000002af8 r12: fffff80160b9c000 r13: fffff87af7163e18 r14: fffff83402622be0 r15: fffff87af7163e00 trap number =3D 12 panic: page fault cpuid =3D 22 time =3D 1771839128 KDB: stack backtrace: #0 0xffffffff8061303d at kdb_backtrace+0x5d #1 0xffffffff805c8091 at vpanic+0x161 #2 0xffffffff805c7f23 at panic+0x43 #3 0xffffffff80972f00 at trap_pfault+0x3e0 #4 0xffffffff8094af68 at calltrap+0x8 #5 0xffffffff8056b750 at closefp_impl+0xd0 #6 0xffffffff80973847 at amd64_syscall+0x117 #7 0xffffffff8094b85b at fast_syscall_common+0xf8 When inspecting it's kernel stack: (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown= .c:405 #2 0xffffffff805c7beb in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:523 #3 0xffffffff805c80e9 in vpanic (fmt=3D0xffffffff809d2ae7 "%s", ap=3Dap@entry=3D0xfffffe071c126c30) at /usr/src/sys/kern/kern_shutdown.c:967 #4 0xffffffff805c7f23 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:891 #5 0xffffffff80972f00 in trap_fatal (frame=3D, eva=3D) at /usr/src/sys/amd64/amd64/trap.c:1000 #6 0xffffffff80972f00 in trap_pfault (frame=3D0xfffffe071c126cb0, usermode=3Dfalse, signo=3D, ucode=3D) #7 #8 0xffffffff80572e28 in knote_drop (kn=3D0xfffff83402622be0, td=3D0xfffff80160b9c000) at /usr/src/sys/kern/kern_event.c:2730 #9 knote_fdclose (td=3D0xfffff80160b9c000, fd=3D201569) at /usr/src/sys/kern/kern_event.c:2695 #10 0xffffffff8056b750 in closefp_impl (fdp=3D0xfffffe0d1582a920, fd=3D0, fp=3D0xfffff81090d2c5a0, td=3D0xfffff80160b9c000, audit=3Dtrue) at /usr/src/sys/kern/kern_descrip.c:1320 #11 0xffffffff80973847 in syscallenter (td=3D0xfffff80160b9c000) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:193 #12 amd64_syscall (td=3D0xfffff80160b9c000, traced=3D0) at /usr/src/sys/amd64/amd64/trap.c:1241 #13 #14 0x000000082deed32a in ?? () Backtrace stopped: Cannot access memory at address 0x85d08dbc8 Within `knote_drop` we observe a null pointer access: (kgdb) fr 8 #8 0xffffffff80572e28 in knote_drop (kn=3D0xfffff83402622be0, td=3D0xfffff80160b9c000) at /usr/src/sys/kern/kern_event.c:2730 2730 kn->kn_fop->f_detach(kn); (kgdb) l 2725 static void 2726 knote_drop(struct knote *kn, struct thread *td) 2727 { 2728=20=20=20=20 2729 if ((kn->kn_status & KN_DETACHED) =3D=3D 0) 2730 kn->kn_fop->f_detach(kn); 2731 knote_drop_detached(kn, td); 2732 } 2733=20=20=20=20 2734 static void (kgdb) p kn->kn_fop $2 =3D (const struct filterops *) 0x0 If you need more info, please ask. We will be glad to provide it. --------------- System info: FreeBSD frv21.ukr.net 14.4-STABLE FreeBSD 14.4-STABLE stable/14-n273658-2f91ff89c56e FRV21 amd64 1404500 1404500 --=20 You are receiving this mail because: You are the assignee for the bug.=