From nobody Sat Jun 6 00:58:06 2026 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gXKg02fCqz6grBb for ; Sat, 06 Jun 2026 00:58:20 +0000 (UTC) (envelope-from rlibby@gmail.com) Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gXKfz49mXz3nNB for ; Sat, 06 Jun 2026 00:58:19 +0000 (UTC) (envelope-from rlibby@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of rlibby@gmail.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=rlibby@gmail.com Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-7ddd3d67084so28198747b3.3 for ; Fri, 05 Jun 2026 17:58:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780707498; x=1781312298; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=TRwannWoquZ5vjzn6+f9ktVGuymEm/m8jbM8G7x0xjY=; b=eDlEbL4glnjLoh0B7y6D29kvvdqrF6XzMwv9J4vGO4r47Kwgmu3FqeSDe9vjxkxjx9 3LnGpbQZNLXF1jBvFpjkhWOyipvbZbuH2nxm8NkAPREHolm7WbKjAXKL+oiAouUDaTSu wSivr+APZskbi2lPXORHzHO16axSCQNe1h492HFK+vbs9gRJtB2Y4P4pYeAjmu3WeeSU FuZAsFKYHOjkDghf8OZ0YgqrHa42TZOgKWhlNY9CpmxtPvzIz8+/Z0owgHH4EELpnb7+ oB4L074fWRwlCtuXrAqhTY7deZPXwQ3f7rHJdoycpuzoVwlbtEsCAzRq6oMUQ2OfcIBL 2nqw== X-Gm-Message-State: AOJu0Yzti2RFHM8tEXrdEpXYwEYEOBq7pB118nV31dKIsFNeGHu26uXn t6qR+41z2gUbYsgIrcII0EJ/Y7ZmHv286SgRMUX1AYkyPvEdO9UBiib3gxzFIA== X-Gm-Gg: Acq92OHb0M/M/oCGpnpHB1cOe+T8vxtX/3JiBxM9QaNSIHwPgTA8mT2BF85ODG5w4LW Xx7LTuZsgOIfksIT6c7A5xLgurBGCLQ64QvYNerRCWYNUP9u/dnKz0gYQGVtXpstTIuRH8HtuSl W3zBQwStK20H1mABGRbjx9S5Oa+JW/Ao+xLior8xynEStwQWsjG56e4aNy5ygzmZT/XtghbUM8X VW52dFr3mHZr8Kv5lcot6bYh0JibBIq9/lp4mOL2vDXCP4A3x66yfKV6XUPYsOwPdanxsro0xur oKiRkuiCzyf7v48qHQUulGsNmiPxMma5ldAjBcsXjNnXsxXuHgGtoMxZyKB9XA8+VK8gBmiOdic qQTj+NSQSUKT/iM/Y+aLxqbjrpQNB0OW/Cgpnzl3IZU4zf7YCjWmuWUaUbcAX61oFompO4PY/MO 66Dn7l0QyUEj8AJPzoDGYGe0kIqSPC47KtLb/ZEkX10cxelme3Wwn4YjpBvShLhEOUzPmZrEwgk OAAqEA= X-Received: by 2002:a05:690c:4909:b0:7d1:eb6c:99bc with SMTP id 00721157ae682-7ed0bf48f5emr71303187b3.11.1780707498522; Fri, 05 Jun 2026 17:58:18 -0700 (PDT) Received: from mail-yx1-f45.google.com (mail-yx1-f45.google.com. [74.125.224.45]) by smtp.gmail.com with ESMTPSA id 00721157ae682-7ea23ea35e1sm57417137b3.43.2026.06.05.17.58.18 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 05 Jun 2026 17:58:18 -0700 (PDT) Received: by mail-yx1-f45.google.com with SMTP id 956f58d0204a3-6611669cd16so536072d50.0 for ; Fri, 05 Jun 2026 17:58:18 -0700 (PDT) X-Received: by 2002:a53:d013:0:b0:651:b477:71cf with SMTP id 956f58d0204a3-66106f5642amr5382845d50.31.1780707497761; Fri, 05 Jun 2026 17:58:17 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 References: <1064242-5q8-qqn-1769-634p9qrropro@mnoonqbm.arg> In-Reply-To: <1064242-5q8-qqn-1769-634p9qrropro@mnoonqbm.arg> From: Ryan Libby Date: Fri, 5 Jun 2026 17:58:06 -0700 X-Gmail-Original-Message-ID: X-Gm-Features: AVHnY4Kfs2q-P5j4fXWWCCWENuq35ExVSX8ZpXDSrMKDSQFRXtKRfFp6Jf7sFqo Message-ID: Subject: Re: Fatal trap 12: .. cpu_idle_acpi .. callout_process To: "Bjoern A. Zeeb" Cc: current@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-2.43 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.53)[-0.535]; FORGED_SENDER(0.30)[rlibby@freebsd.org,rlibby@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[current@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[rlibby@freebsd.org,rlibby@gmail.com]; FREEFALL_USER(0.00)[rlibby]; MLMMJ_DEST(0.00)[current@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.128.181:from]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.128.181:from,74.125.224.45:received] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4gXKfz49mXz3nNB On Tue, Jun 2, 2026 at 9:25=E2=80=AFPM Bjoern A. Zeeb wrote: > > On Wed, 27 May 2026, Bjoern A. Zeeb wrote: > > > On Tue, 26 May 2026, Bjoern A. Zeeb wrote: > > > >> Hi, > >> > >> I got some LinuxKPI problems sorted and can finally shutdown a system = w/o > >> a driver panicing but now I see on a recent main (pxe booted in bhyve)= ; > >> this seems reproducible and typing reset I get the next panic and the = next > >> and the next and ... until bhyve stops after scrolling for a few secon= ds. > >> > >> Anyone seen this or any ideas? I'll try to build a plain main kernel > >> otherwise > >> to check that it's not anything else... > > > > I have already found the next LinuxKPI bug. > > > > If I just boot a kernel and do a shutdown -r I do not run into it > > so unless it rings a bell for someone else as well, please ignore this = for > > now. > > It just happened again; no known LinuxKPI bugs in the way this time. > > So maybe it's real after all... > > > >> Syncing disks, vnodes remaining... 0 done > >> All buffers synced. > >> Uptime: 46s > >> kernel trap 12 with interrupts disabled > >> > >> > >> Fatal trap 12: page fault while in kernel mode > >> cpuid =3D 0; apic id =3D 00 > >> fault virtual address =3D 0xfffffe00a58a0630 > >> fault code =3D supervisor read data, page not present > >> instruction pointer =3D 0x20:0xffffffff80c0ebe8 > >> stack pointer =3D 0x28:0xfffffe008bc49bb0 > >> frame pointer =3D 0x28:0xfffffe008bc49c20 > >> code segment =3D base 0x0, limit 0xfffff, type 0x1b > >> =3D DPL 0, pres 1, long 1, def32 0, gran 1 > >> processor eflags =3D resume, IOPL =3D 0 > >> current process =3D 11 (idle: cpu0) > >> rdi: 0000000000002f2c rsi: 0000000000008000 rdx: 0000000000002e2d > >> rcx: 0000000000002e2c r8: fffffe00a58a0630 r9: 000000007fff2744 > >> rax: fffffe000ef4e000 rbx: 0000000000002e2c rbp: fffffe008bc49c20 > >> r10: 00000000000003e7 r11: 000000000000044c r12: 0000002f2d000000 > >> r13: 0000002f2d000000 r14: 0000002e2dd1597a r15: ffffffff82b28300 > >> trap number =3D 12 > >> panic: page fault > >> cpuid =3D 0 > >> time =3D 1779819492 > >> KDB: stack backtrace: > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame > >> 0xfffffe008bc498e0 > >> vpanic() at vpanic+0x149/frame 0xfffffe008bc49a10 > >> panic() at panic+0x43/frame 0xfffffe008bc49a70 > >> trap_pfault() at trap_pfault+0x449/frame 0xfffffe008bc49ae0 > >> calltrap() at calltrap+0x8/frame 0xfffffe008bc49ae0 > >> --- trap 0xc, rip =3D 0xffffffff80c0ebe8, rsp =3D 0xfffffe008bc49bb0, = rbp =3D > >> 0xfffffe008bc49c20 --- > >> callout_process() at callout_process+0x138/frame 0xfffffe008bc49c20 > >> handleevents() at handleevents+0x19a/frame 0xfffffe008bc49c60 > >> timercb() at timercb+0x19e/frame 0xfffffe008bc49cc0 > >> lapic_handle_timer() at lapic_handle_timer+0xa4/frame 0xfffffe008bc49c= f0 > >> Xtimerint() at Xtimerint+0xb1/frame 0xfffffe008bc49cf0 > >> --- interrupt, rip =3D 0xffffffff810b1104, rsp =3D 0xfffffe008bc49dc0,= rbp =3D > >> 0xfffffe008bc49dd0 --- > >> cpu_idle_acpi() at cpu_idle_acpi+0x54/frame 0xfffffe008bc49dd0 > >> cpu_idle() at cpu_idle+0xa6/frame 0xfffffe008bc49df0 > >> sched_ule_idletd() at sched_ule_idletd+0x524/frame 0xfffffe008bc49ef0 > >> fork_exit() at fork_exit+0x82/frame 0xfffffe008bc49f30 > >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008bc49f30 > >> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- > >> KDB: enter: panic > >> [ thread pid 11 tid 100003 ] > >> Stopped at kdb_enter+0x33: movq $0,0x15be0c2(%rip) > >> db> reset > >> panic: mtx_lock_spin: recursed on non-recursive mutex callout @ > >> /usr/src/sys/kern/kern_timeout.c:576 > >> > >> cpuid =3D 0 > >> time =3D 1779819492 > >> KDB: stack backtrace: > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame > >> 0xfffffe008bc49160 > >> vpanic() at vpanic+0x149/frame 0xfffffe008bc49290 > >> panic() at panic+0x43/frame 0xfffffe008bc492f0 > >> __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x11b/frame > >> 0xfffffe008bc49330 > >> _callout_stop_safe() at _callout_stop_safe+0x106/frame 0xfffffe008bc49= 3a0 > >> shutdown_resettodr() at shutdown_resettodr+0x15/frame 0xfffffe008bc493= b0 > >> kern_reboot() at kern_reboot+0x2a3/frame 0xfffffe008bc493f0 > >> db_reset() at db_reset+0x108/frame 0xfffffe008bc49420 > >> db_command() at db_command+0x3aa/frame 0xfffffe008bc494e0 > >> db_command_loop() at db_command_loop+0x4d/frame 0xfffffe008bc494f0 > >> db_trap() at db_trap+0x100/frame 0xfffffe008bc49590 > >> kdb_trap() at kdb_trap+0x25f/frame 0xfffffe008bc496e0 > >> trap() at trap+0x888/frame 0xfffffe008bc49810 > >> calltrap() at calltrap+0x8/frame 0xfffffe008bc49810 > >> --- trap 0x3, rip =3D 0xffffffff80c44f43, rsp =3D 0xfffffe008bc498e8, = rbp =3D > >> 0xfffffe008bc49a10 --- > >> kdb_enter() at kdb_enter+0x33/frame 0xfffffe008bc49a10 > >> panic() at panic+0x43/frame 0xfffffe008bc49a70 > >> trap_pfault() at trap_pfault+0x449/frame 0xfffffe008bc49ae0 > >> calltrap() at calltrap+0x8/frame 0xfffffe008bc49ae0 > >> --- trap 0xc, rip =3D 0xffffffff80c0ebe8, rsp =3D 0xfffffe008bc49bb0, = rbp =3D > >> 0xfffffe008bc49c20 --- > >> callout_process() at callout_process+0x138/frame 0xfffffe008bc49c20 > >> handleevents() at handleevents+0x19a/frame 0xfffffe008bc49c60 > >> timercb() at timercb+0x19e/frame 0xfffffe008bc49cc0 > >> lapic_handle_timer() at lapic_handle_timer+0xa4/frame 0xfffffe008bc49c= f0 > >> Xtimerint() at Xtimerint+0xb1/frame 0xfffffe008bc49cf0 > >> --- interrupt, rip =3D 0xffffffff810b1104, rsp =3D 0xfffffe008bc49dc0,= rbp =3D > >> 0xfffffe008bc49dd0 --- > >> cpu_idle_acpi() at cpu_idle_acpi+0x54/frame 0xfffffe008bc49dd0 > >> cpu_idle() at cpu_idle+0xa6/frame 0xfffffe008bc49df0 > >> sched_ule_idletd() at sched_ule_idletd+0x524/frame 0xfffffe008bc49ef0 > >> fork_exit() at fork_exit+0x82/frame 0xfffffe008bc49f30 > >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008bc49f30 > >> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- > >> panic: mtx_lock_spin: recursed on non-recursive mutex callout @ > >> /usr/src/sys/kern/kern_timeout.c:576 > >> > >> cpuid =3D 0 > >> time =3D 1779819492 > >> .. > >> .. > >> .. > >> > >> > >> > > > > > > -- > Bjoern A. Zeeb r15:7 > Can you resolve this? > callout_process() at callout_process+0x138 Just guessing from my local kernel, that may be the first touch of a callout in the LIST_FOREACH_SAFE loop of callout_process. If so that may suggest a use after free of some callout, with a dangling pointer to the callout remaining in the list. Maybe someone freed some callout without stopping it. Or maybe the list is corrupt in some other way. Ryan