From nobody Tue Nov 4 15:41:06 2025 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d1CNQ1289z6GGk1 for ; Tue, 04 Nov 2025 15:41:10 +0000 (UTC) (envelope-from mhorne@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d1CNQ0YQSz3Sh5; Tue, 04 Nov 2025 15:41:10 +0000 (UTC) (envelope-from mhorne@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1762270870; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=26bTjGVM5RU+VopwRfKBHW4hyLj7czWHGfB4ly3jNrA=; b=DOIW8gIJ4urgfeab7hyQS3CpZq3MP0TA/KfstywH6hDMeyXGtkWE1WbfiZNYdkiM4Oc/fG 7G36Hg3sTvecCYnOigiP6S4jFlb7UlDU0C6wdUDw/1mavoQgFPs6tOboxykx7PSxqSnxMa pvk3prqqQ+sTeWkkvCxsvGAjFciktO8L9LoQ2XFfnF5RE/gTMfwswnrDzBT4MxDbgzbmRy 5/8xbRsQ4bWFYabx8TW3gN+jXAr8L13isrlNLjing1ncC67GLY87DFEOtLCOwBjEidTaQH jYw+XBwove7JqnYJLQ03riwbhkOotJ71TPouAqyYVPqkZjVNjqGZqC9c/QuTqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1762270870; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=26bTjGVM5RU+VopwRfKBHW4hyLj7czWHGfB4ly3jNrA=; b=VHrnB1mghr8u/mEBhstQ3NEkqFBOOy8ZvkQxRPFu6UhdM/Ihqxd5zDJmvAO4JW21wYMXEH Nb1Pmj2/rIFEA4gbLlqAcX/68VKJyXDf0C3tvIX17CdpkNT6eOuWisSyn2CU+zqyNrbiX4 BE/+AI9g/7Ens0n7FJt9B/OIdY3nEERUFT0ETp4vfq6LUzvNKemHZ4gvBDDtVn5TIrIIpK YI6yP0y52YkG/TZa1sR806SBOxcV7nT3sWdkxPEKkDHWprT0o2sscNhN5QvlX4edgcelBy NB//KsHOy1z4puhEipmIM7T2Ex9QrH6JFEVvdchl6aqyUzBqoVvAVFGIpkjZcw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1762270870; a=rsa-sha256; cv=none; b=pKnB9Vo/A5WGqrhi9sc0i+uVSgeiiDfDI2ziRzlhiEINiwSF+tH58Wo+nAMxkTBNI8a8zk Uv/krwE13a4XWHN/iZL4pQswuFFu9bvec/I0aG548uquQ8+DNPjGjJ8p3tkve4Ek4gqXbN lGbuGfLcc7KAsXRYct2pyIrBf8W1du4/iivhQvzgFYGapzo92QENefUi5fB3vkty87hoqr VgyfZZI6y4rfVHFark5kaj+wrM+ttfEDb8H0QI8hpCKqNOqKbpOB2H9Di1iFkAvO9/2gxN 7FNfdMQbejCBybY+uFcXOhAbZSuuJvsquuvXI4kABqzxcrtU/o/BC6qMin/y6A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [192.168.2.224] (unknown [142.68.228.197]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mhorne) by smtp.freebsd.org (Postfix) with ESMTPSA id 4d1CNP65bdzQZP; Tue, 04 Nov 2025 15:41:09 +0000 (UTC) (envelope-from mhorne@freebsd.org) Message-ID: <2184dc51-3ab9-479c-be24-3bbd7f2c8809@freebsd.org> Date: Tue, 4 Nov 2025 11:41:06 -0400 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: db> reset -> panic: lock (sleep mutex) eventhandler not locked @ /usr/src/sys/kern/subr_eventhandler.c:272 To: Ryan Libby , "Bjoern A. Zeeb" Cc: current@freebsd.org References: Content-Language: en-CA From: Mitchell Horne In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hi both, On 10/28/25 06:59, Ryan Libby wrote: > On Tue, Oct 28, 2025 at 2:46 AM Ryan Libby wrote: >> >> On Mon, Oct 27, 2025 at 7:31 AM Bjoern A. Zeeb >> wrote: >>> >>> Hi, >>> >>> on main-ish I get the following. I am a bit concerned as over the last >>> year or two our reset paths had issues like this more and more. >>> >>> >>> db> reset >>> panic: lock (sleep mutex) eventhandler not locked @ /usr/src/sys/kern/subr_eventhandler.c:272 >>> cpuid = 1 >>> time = 1025 >>> KDB: stack backtrace: >>> db_trace_self() at db_trace_self >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x38 >>> vpanic() at vpanic+0x1a0 >>> panic() at panic+0x48 >>> witness_unlock() at witness_unlock+0x140 >>> __mtx_unlock_flags() at __mtx_unlock_flags+0x54 >>> eventhandler_find_list() at eventhandler_find_list+0xbc >>> kern_reboot() at kern_reboot+0x244 >>> db_reset() at db_reset+0xec >>> db_command() at db_command+0x2f4 >>> db_command_loop() at db_command_loop+0x58 >>> db_trap() at db_trap+0x100 >>> kdb_trap() at kdb_trap+0x350 >>> handle_el1h_sync() at handle_el1h_sync+0x18 >>> --- exception, esr 0xf2000000 >>> kdb_alt_break_internal() at kdb_alt_break_internal+0x1a8 >>> kdb_alt_break() at kdb_alt_break+0x10 >>> uart_intr_rxready() at uart_intr_rxready+0x88 >>> uart_intr() at uart_intr+0x124 >>> intr_event_handle() at intr_event_handle+0xf4 >>> intr_isrc_dispatch() at intr_isrc_dispatch+0x60 >>> arm_gic_intr() at arm_gic_intr+0x118 >>> intr_irq_handler() at intr_irq_handler+0x98 >>> handle_el1h_irq() at handle_el1h_irq+0x18 >>> --- interrupt >>> cpu_idle() at cpu_idle+0x78 >>> sched_idletd() at sched_idletd+0x494 >>> fork_exit() at fork_exit+0x78 >>> fork_trampoline() at fork_trampoline+0x18 >>> Uptime: 17m5s >>> Automatic reboot in 15 seconds - press a key on the console to abort >>> >>> -- >>> Bjoern A. Zeeb r15:7 >>> Bjoern, I apologize that this change once again is hindering the reset path. I share your concern, as do others, please see the renewed discussion for a possible path forward: https://reviews.freebsd.org/D37981 >> >> Would your kernel config match this condition in sys/sys/mutex.h? >> #if LOCK_DEBUG > 0 || defined(MUTEX_NOINLINE) >> >> It looks like we might be missing some SCHEDULER_STOPPED() checks in >> those code paths. It's a twisty maze of macros that I haven't totally >> followed though. td_locks might be getting broken too. >> >> Ryan > > Reading sys/sys/_lock.h, LOCK_DEBUG will be on with WITNESS, which this > kernel clearly has (and also with INVARIANTS, both of which are defaults > for GENERIC). > Ryan, thanks for your analysis. This one in particular was known to me, and I've just put the change up in phabricator. As best I can tell this is all that is missing... https://reviews.freebsd.org/D53582 Best, Mitchell