Date: Mon, 10 Jan 2022 19:11:10 -0500 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: Mateusz Guzik <mjguzik@gmail.com> Cc: Mark Johnston <markj@freebsd.org>, freebsd-hackers@freebsd.org Subject: Re: Debugging a (potentially?) ZFS-related panic, and discussion about large patchsets Message-ID: <20220111001110.medkloif6zghtatg@mutt-hbsd> In-Reply-To: <20220111001023.wx5nh64a5zqq7cae@mutt-hbsd> References: <20220110221116.gustgfgfge6pb5fe@mutt-hbsd> <YdzCatNYBVDFi9So@nuc> <CAGudoHHuNbYNFsc3vqihFkQ0xA9Hojp21-QuHczVXvMVSHfQ5w@mail.gmail.com> <20220111001023.wx5nh64a5zqq7cae@mutt-hbsd>
next in thread | previous in thread | raw e-mail | index | archive | help
--vbzw4udfy3tf73no
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, Jan 10, 2022 at 07:10:23PM -0500, Shawn Webb wrote:
> On Tue, Jan 11, 2022 at 12:43:06AM +0100, Mateusz Guzik wrote:
> > On 1/11/22, Mark Johnston <markj@freebsd.org> wrote:
> > > On Mon, Jan 10, 2022 at 05:11:16PM -0500, Shawn Webb wrote:
> > >> Hey all,
> > >>
> > >> So I'm getting an interesting ZFS-related kernel panic. I've uploaded
> > >> the core.txt at [0]. I suspect it's related to FreeBSD commit
> > >> 681ce946f33e75c590e97c53076e86dff1fe8f4a (zfs: merge
> > >> openzfs/zfs@f291fa658 (master) into main).
> > >>
> > >> I'm able to reproduce it on a single system with some level of
> > >> determinism: I'm building the security appliance firmware at ${DAYJO=
B}
> > >> in a bhyve VM that's backed by a zvol. The host is a Dell Precision
> > >> 7540 laptop with a single NVMe drive in it. The VM is configured with
> > >> a single zvol, booting with UEFI.
> > >>
> > >> Looking at the commit email sent to dev-commits-src-all@, I see this:
> > >> 146 files changed, 4933 insertions(+), 1572 deletions(-)
> > >>
> > >> Strangely, when I run `git show
> > >> 681ce946f33e75c590e97c53076e86dff1fe8f4a`, I only see a small subset
> > >> of those changes.
> > >
> > > That is a merge commit. You need to specify that you want a diff
> > > against the first parent (the preceding FreeBSD), so something
> > > equivalent to "git diff --stat 681ce946f^ 681ce946f". Use
> > > "git log 681ce946f^2" to see the merged OpenZFS commits.
> > >
> > >> As a downstream consumer of 14-CURRENT, how am I supposed to even
> > >> start debugging such a large patchset in any manner that respects my
> > >> time?
> > >>
> > >> It seems to me that breaking up commits into smaller, bite-size chun=
ks
> > >> would make life easier for those experiencing bugs, especially ones
> > >> that result in kernel panics.
> > >
> > > That's up to the upstream project, in this case OpenZFS.
> > >
> > >> ZFS in and of itself is a beast, and I've yet to study any of its
> > >> code, so when there's a commit that large, even thinking about
> > >> debugging it is a daunting task.
> > >>
> > >> Needless to say, I'm going to need some hand holding here for
> > >> debugging this. Anyone have any idea what's going on?
> > >
> > > To start, you'll need to look at the stack trace for the thread with =
tid
> > > 100061.
> > >
> >=20
> > imo the kernel should be patched to obtain the trace on its own. As
> > the target has interrupts disabled it will have to do it with NMI, but
> > support for that got scrapped in
> >=20
> > commit 1c29da02798d968eb874b86221333a56393a94c3
> > Author: Mark Johnston <markj@FreeBSD.org>
> > Date: Fri Jan 31 15:43:33 2020 +0000
> >=20
> > Reimplement stack capture of running threads on i386 and amd64.
>=20
> I guess it's especially problematic for laptop systems where dropping
> to the db> prompt isn't an option (nvidia driver on this laptop). I'd
> have to scrap the entire notion of a GUI, which kinda defeats the
> purpose of using a laptop.
>=20
> Plugging in a USB memstick and setting debug.trace_on_panic=3D0 is the
> route I usually take on such systems.
Sorry, wrong sysctl node. I meant to reference
debug.debugger_on_panic.
--=20
Shawn Webb
Cofounder / Security Engineer
HardenedBSD
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A=
4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
--vbzw4udfy3tf73no
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmHcyx4ACgkQ/y5nonf4
4fpWrw/+IdLPNu9IUlQaq/iXCbhIP4+oCYA/k3KyhjDW6HL77Il2G4dwMbJ+UVGU
QEhBAmPi8Ba3JKzxxy6GQwMff2gkC2ieWCltb3VEdzUvx38inyxzjxRCJ76oljTF
CpoFvnRKGsl4yGOwJnhgtCogH1fkB6r2N5CTQ23axTYqops6bAt0SAPkdcXMih8B
furpjFJElYBm8VUp1RQ53DbTcTALH6wxWqFYjX/yvlAcxQAPJLPmBy2H7WNXN6uX
/ynpTleQW5xzY0emL2aB7J2dnw6fCzqC4Rorj6Qw0ipw8fd+3/JNdQbpPlIWwBlp
RSdBbsi2MJvay+jFkX3IEW/eKZWBHHwHGHo1sbgDCc4FKWft6PyPooraVMMjMjhN
BhxM8XyRWCMgdXbQHq5K4WLvqdZxmcPBTalBshPmLPNnL5CcifPqVQrH6xaO/JNP
0IiV6ilbUXJIDb5sIeT7tSiGyV0Jo8Bhoh+lVc6LPygqHYW6ujcayELTpTY/JrNJ
q/AikJ2TiYLV+qBa1VfflGF2y2T0v0itEapN1QzNuA6HKbvABOljd/v5wR6hkHkG
rRQHA2UtjVgVfnqiSo5wT1vTD7eagy/rfgwNUQ92OGJcFXRdlT6mTP3CUbRTH3pz
zT+wXcjISqZzdkTHcMOZu2O8isLfPRcQY9T5vyIg3Iz/QEBh3eM=
=274Z
-----END PGP SIGNATURE-----
--vbzw4udfy3tf73no--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20220111001110.medkloif6zghtatg>