From nobody Tue Jul 4 15:06:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QwR2H1hRFz4lrMR for ; Tue, 4 Jul 2023 15:06:15 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QwR2H11DZz3sbF; Tue, 4 Jul 2023 15:06:15 +0000 (UTC) (envelope-from otis@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688483175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R3cSb9hXXwRRDIgqJmQd8pu2N3fD3wF6clOlHiW8+3U=; b=OKtCSh/Fiyt5Ny43bpJx47g+xZn/Cj0KKqjvLYSKIpIsTCE4pUpVOxkdr1m2uybuzeGe8x AvSdOi46djeLkKz3lc58Bk/f/iCVhirP1KCCwhLUFM1XeC2xFlKp15h1qPpZDI25bek1D3 MA9HHoISFJb/FaRpe5hQWq7o0RE08O5Hc9TA8kH6AStVK3542ufOi5SIH699EQu7yIEWMq AiU00S75GGci557Bl4Yt0ZqHspSitv7XIKwqwgCBEzdrLT4xE3KXR9Lc+fXTxDS9lY/mIh P1WMQ7L6q8KnvmXRE+r0NCKEe30FHuMvNOM3KUUd4zoZkrQreoT3wWMHtsFJaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1688483175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R3cSb9hXXwRRDIgqJmQd8pu2N3fD3wF6clOlHiW8+3U=; b=A2zf4p5aGuIhc0TVGXrbg4ASBMtyHxJ4YQ770sY+C5B/bUYNf5m+UfKisZODM1n2xX/b1O oTwpdX7C0VhPneQsz9/Su0uJAcPu/I+QOldnLEfuQwBzFz5uEcY21bGr8M9+NOYH4L+Rhb 8rs/j21HlkODlbsOpaS6n6WqA5AZX2avmDVHA0W1on5XLc5swEUV/YihnUsBrnUW+zInpp Szye0jeCMlHN8a3+7saFPqi3ESMKJqnKTRJcpgBVsbm1D4JSz14fWYauUIqtQpootCAFAl ZCnKkGZ3FiDFIxHbrjyoLiH29H8QnGMZTZFhOZBIKbtk4ya36VcuNTHzV1UDMg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1688483175; a=rsa-sha256; cv=none; b=oj6D+edXlgMK+sRuIRUFLpglyERnVuStIzntfX2iV81KQoA6nAaURVSNCaAGTDbZt+mi6R luU7GZ/wVNPbW3MB514Gf/TJJZg1ZsGsCt/CNoJQfHsFm9fySnihUqlkbxz3AN1ht/RqkV cxE0CF3xRM62D0HLLLWCPW5Y1L1wkiMFwSus64oFVfKdOf9hxg6gciWE80q+dmGoAKXcTL QaauMfzI2eiV2DEJyp11gwt9Gbfd29fvdNmmRS6l/ag4HqpOfuoYcY0pyXHQzdvIXs6Bvo b7jZ5u3aeOqoSE4FS+KazTlqP7Lo8KcUkPMHdImv+p5pZyQEHYp/XUtUnAVf8g== Received: from ns2.wilbury.net (ns2.wilbury.net [92.60.51.55]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "svc.wilbury.net", Issuer "R3" (verified OK)) (Authenticated sender: otis) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QwR2G6M9TzJvR; Tue, 4 Jul 2023 15:06:14 +0000 (UTC) (envelope-from otis@FreeBSD.org) Received: from smtpclient.apple (unknown [217.73.28.193]) (Authenticated sender: juraj@lutter.sk) by svc.wilbury.net (Postfix) with ESMTPSA id 36EC761F85; Tue, 4 Jul 2023 17:06:11 +0200 (CEST) Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: nvme related(?) panic on recent -CURRENT From: Juraj Lutter In-Reply-To: Date: Tue, 4 Jul 2023 17:06:00 +0200 Cc: FreeBSD CURRENT Content-Transfer-Encoding: quoted-printable Message-Id: <3A5E3ACD-9BBF-4A4E-BE06-8B3D300E9046@FreeBSD.org> References: <664978DE-B510-4C14-A559-541C3AB00D50@FreeBSD.org> To: Chuck Tuffli X-Mailer: Apple Mail (2.3731.600.7) X-Spam-Status: No, score=-2.8 required=5.0 tests=ALL_TRUSTED,BAYES_00,TW_NV, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on ns2.wilbury.net X-ThisMailContainsUnwantedMimeParts: N > On 4 Jul 2023, at 17:01, Chuck Tuffli wrote: >=20 > On Thu, Jun 29, 2023 at 12:47=E2=80=AFPM Juraj Lutter = wrote: >>=20 >> With recent -current, following occured: >>=20 >> db> bt >> Tracing pid 0 tid 100063 td 0xfffffe00c5c35e40 >> kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c5e31c90 >> vpanic() at vpanic+0x181/frame 0xfffffe00c5e31ce0 >> panic() at panic+0x43/frame 0xfffffe00c5e31d40 >> nvme_ctrlr_identify() at nvme_ctrlr_identify+0x10e/frame = 0xfffffe00c5e31d90 >> nvme_ctrlr_start() at nvme_ctrlr_start+0x91/frame 0xfffffe00c5e31e10 >> nvme_ctrlr_reset_task() at nvme_ctrlr_reset_task+0xec/frame = 0xfffffe00c5e31e40 >> taskqueue_run_locked() at taskqueue_run_locked+0x182/frame = 0xfffffe00c5e31ec0 >> taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame = 0xfffffe00c5e31ef0 >> fork_exit() at fork_exit+0x7d/frame 0xfffffe00c5e31f30 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c5e31f30 >> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 =E2=80=94 >>=20 >> machine is a bhyve guest. >=20 > If I'm lldb'ing correctly, nvme_ctrlr_identify+0x10e is the panic in > nvme_completion_poll() if the NVMe command does not complete within > the timeout period (10 seconds). In this case, it is the Identify, > Controller command. In the bhyve emulation, this command effectively > memcpy's the data structure to the memory provided by the guest and > completes the command. If this panic is reproducible, I can provide a > patch to enhance the debug output to figure out if this is an > emulation or driver issue. It hasn=E2=80=99t happened since. What I can do is to put heavy load on that box (poudriere in a jail as well as poudriere in that VM). That could help to reproduce the panic. As mentioned somewhere else in this thread, I=E2=80=99ve a `cu=E2=80=99 session running. otis =E2=80=94 Juraj Lutter otis@FreeBSD.org