From nobody Tue Jan 11 00:10:23 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 755BA193026C for ; Tue, 11 Jan 2022 00:10:25 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JXrgP2BH7z3M8B for ; Tue, 11 Jan 2022 00:10:25 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt1-x829.google.com with SMTP id h4so5009111qth.11 for ; Mon, 10 Jan 2022 16:10:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=pHUBAotWHB5SvHyVKIqg+mpc95LrWAxD/z2m0AQ3Q94=; b=LG292cRWzsF8HZQbwWGsaYco9ReyGTvT9mYka1c3aWVX2iltQ/wr4CGsxHzNae8ot/ oMTNSHxRKzFUx8WFooiBZGjq4tN/vAKuCtV458iPFVVTMPD9vMGAQe6ns7vTcqnBf1Mm S8QufcIwVMNuVDaxsIfUVRNKKImQXBJnFun8SLqAWMgck8RuMM9zzx33nV7zgVAIJ85c OX0umC07jOuebu9RuwOMrqSHzl45UCXEOFBbI0BFo9lGPKBo/zbRrHtoWd3BXH4AWeY9 j5fNddWnuxU+mr3+s5Cxi92o2we/5QsyUz5kRPKAT0Ue/sGFoTgZ3aar6bDWf+E/QWCm WTvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=pHUBAotWHB5SvHyVKIqg+mpc95LrWAxD/z2m0AQ3Q94=; b=efeB1RXllyzmEnjRdF1qw9//THJLWFYipCIwZVmJLb39MK6bIpjH7kByC66P4dgqzq BKK8mNmyTUhV5IBbWe81Nd4UlrOw/RaACmphADFeQBA1uM2CvKNyM9UC8wrrXs8lQ/gt B2eNcIhVCEbjUuR+s8U1Juvc+GovIteg5cYCmC59IqQcGRL8yFN44cJpLmUbSnJn+LyL U9JsFyagV6VCR2mTatOOET00VzxoOmYYVZnwgbrofrzHqQbdrVNsNBzvH0x0PZbUtZ0l SPrYESmtrspzy6zs3HkguBgzty3ESVhFLjYs2ZnxCma6B6BqBQ72g5Uqr1sJnwGl23J9 Yyiw== X-Gm-Message-State: AOAM5324OLbQm6b2sMjD7YmmHVmOgcERV3CPQHqQ4IJLi8Uxatb4KT0x TcEYmQhmdx5mHqUBb2klOed1gA== X-Google-Smtp-Source: ABdhPJxGFdWGQjRz+4CBbfPcmOf0WIU2SO+SWxL/PBESS2PEYur0xcD7XdQo+Yi/ibFw+x4EJok09A== X-Received: by 2002:ac8:57d1:: with SMTP id w17mr1891847qta.453.1641859824745; Mon, 10 Jan 2022 16:10:24 -0800 (PST) Received: from mutt-hbsd (pool-100-16-224-136.bltmmd.fios.verizon.net. [100.16.224.136]) by smtp.gmail.com with ESMTPSA id u9sm5171762qkp.116.2022.01.10.16.10.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 16:10:24 -0800 (PST) Date: Mon, 10 Jan 2022 19:10:23 -0500 From: Shawn Webb To: Mateusz Guzik Cc: Mark Johnston , freebsd-hackers@freebsd.org Subject: Re: Debugging a (potentially?) ZFS-related panic, and discussion about large patchsets Message-ID: <20220111001023.wx5nh64a5zqq7cae@mutt-hbsd> X-Operating-System: FreeBSD mutt-hbsd 14.0-CURRENT-HBSD FreeBSD 14.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: <20220110221116.gustgfgfge6pb5fe@mutt-hbsd> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="um6yum3kqmcg5s5c" Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4JXrgP2BH7z3M8B X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N --um6yum3kqmcg5s5c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 11, 2022 at 12:43:06AM +0100, Mateusz Guzik wrote: > On 1/11/22, Mark Johnston wrote: > > On Mon, Jan 10, 2022 at 05:11:16PM -0500, Shawn Webb wrote: > >> Hey all, > >> > >> So I'm getting an interesting ZFS-related kernel panic. I've uploaded > >> the core.txt at [0]. I suspect it's related to FreeBSD commit > >> 681ce946f33e75c590e97c53076e86dff1fe8f4a (zfs: merge > >> openzfs/zfs@f291fa658 (master) into main). > >> > >> I'm able to reproduce it on a single system with some level of > >> determinism: I'm building the security appliance firmware at ${DAYJOB} > >> in a bhyve VM that's backed by a zvol. The host is a Dell Precision > >> 7540 laptop with a single NVMe drive in it. The VM is configured with > >> a single zvol, booting with UEFI. > >> > >> Looking at the commit email sent to dev-commits-src-all@, I see this: > >> 146 files changed, 4933 insertions(+), 1572 deletions(-) > >> > >> Strangely, when I run `git show > >> 681ce946f33e75c590e97c53076e86dff1fe8f4a`, I only see a small subset > >> of those changes. > > > > That is a merge commit. You need to specify that you want a diff > > against the first parent (the preceding FreeBSD), so something > > equivalent to "git diff --stat 681ce946f^ 681ce946f". Use > > "git log 681ce946f^2" to see the merged OpenZFS commits. > > > >> As a downstream consumer of 14-CURRENT, how am I supposed to even > >> start debugging such a large patchset in any manner that respects my > >> time? > >> > >> It seems to me that breaking up commits into smaller, bite-size chunks > >> would make life easier for those experiencing bugs, especially ones > >> that result in kernel panics. > > > > That's up to the upstream project, in this case OpenZFS. > > > >> ZFS in and of itself is a beast, and I've yet to study any of its > >> code, so when there's a commit that large, even thinking about > >> debugging it is a daunting task. > >> > >> Needless to say, I'm going to need some hand holding here for > >> debugging this. Anyone have any idea what's going on? > > > > To start, you'll need to look at the stack trace for the thread with tid > > 100061. > > >=20 > imo the kernel should be patched to obtain the trace on its own. As > the target has interrupts disabled it will have to do it with NMI, but > support for that got scrapped in >=20 > commit 1c29da02798d968eb874b86221333a56393a94c3 > Author: Mark Johnston > Date: Fri Jan 31 15:43:33 2020 +0000 >=20 > Reimplement stack capture of running threads on i386 and amd64. I guess it's especially problematic for laptop systems where dropping to the db> prompt isn't an option (nvidia driver on this laptop). I'd have to scrap the entire notion of a GUI, which kinda defeats the purpose of using a laptop. Plugging in a USB memstick and setting debug.trace_on_panic=3D0 is the route I usually take on such systems. Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --um6yum3kqmcg5s5c Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmHcyu0ACgkQ/y5nonf4 4fqSXQ/+MXeioqkK8F9/nKElYxQg/JhM5FnlY/hhgqrxI1wzF+pYCTU6eyZUWMKK 0a8xC+ws+KMNmpZRQQfQiXTyaLoVP2xR5rMF5IYhX3/Q7CdC97gWuY1NV+AJtmBm I5UXhPBp+hoTxK9DNnzMtw7xDTCDTCDdNCxZikOtUxlrWuFdFqhku5cGE4psj3A/ dmaqSEc6bA34JKbSm/vnOZrF4fMZ7Mtd4d6BTFNNrumT0HT9VfvLykaxIDUz624W Z+3LuLwnN6yXeRwxMW+90MigYk7U38MIpxR8IzMHCPZS7vilMh5FzYcxXNvEP+wG WhnZUbSBZYfSeLsXdGHSB3akQ5Ro0Ev7badeFBUF1g0vVS3WJPMAF+pyTT04Jo6v UAQ6roDKzywQZ0k+1H7F8Tkx+4OJlluttx8WbIUSufpX459PQYs+V+ZQ3vh+tyLj G7UZsbgrmJAsTOhqyJ556L5Cra+lUbu1uGau0e+SOCwHINHU3NicGonV5dLdsX6W QPt9DpK86rnJfIIPmAfuh210jcm8hr9BsGVj3zm8obfNCg5UROiqMSG0Kr6XssS7 SYmc9cjFfYooxyHN1rokw1tMVllf2q19ZgF5z5LGUdnlOaKjHBwMCYNW798xUzC2 xGjWwqUT0JU5Om7LlH1ilik0cbvByaEQwZg9zeHRRefB3Zk4GeU= =y22k -----END PGP SIGNATURE----- --um6yum3kqmcg5s5c--