From nobody Mon Jan 10 23:43:06 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id F41A41950B66 for ; Mon, 10 Jan 2022 23:43:07 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JXr3v6Qkpz3G76; Mon, 10 Jan 2022 23:43:07 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-ot1-x32e.google.com with SMTP id i5-20020a05683033e500b0057a369ac614so16800217otu.10; Mon, 10 Jan 2022 15:43:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=uqNDlW1VOJXhDn6t1u5Uxt5K3gKGRUWeQg4YIgEBv3E=; b=Xv9t7c05NqHRmcbCnusaWny+SlzHB4bdqnytqiQs2Jp/l91jz5nEsfD48t2urhdEsg q0uD6QK79z9hVJB85rTYdCfRDnX6FLcSkbznzRkQxNTB2T4S84kWoqgVEi8EkIo+1HBG hzhVR0sB1CYI/xvht3JZSnMBzP+G+yU930AeIoUrrQ28MHaWKBLwlNvpF+duS3vCXHkG udcGLtn0AVV7foNxx7EPf5uS/X3m3LP3NhbdmhfGtzehcGPS7Mh48k9b+8Y97E0uEAIG BHsAj5HkFtDp8KBC2G2kTZ9RKgagyP+ewSepZnjui7w7sOa7WxmON0J3z1ferw7l2D6T pUSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=uqNDlW1VOJXhDn6t1u5Uxt5K3gKGRUWeQg4YIgEBv3E=; b=X49GGOKfOP1lIgZvOZ9Y77brRUbIrAGXxRbkwYjqIGujd7jQjyYPUJD83XhOnCtUEs DFVeXI3wvSfgZdLpDSQV8466XNkx1Nhsobx4xr9+hG6toOws1E6jN0ixh1QSUPlQbDTR 4wl4zI6IHJNkZm2brwcqIKaHCuqk7U7r5PwR8HY11uYuLc+KWHaGFxT4iND3Wjrzr8j1 iFHXKq71VvNao4rnX0jXzDAtsfr1UO6SnRzSo1Moe4J/BkabRPVgxy21OZCVl4HKoebV E/uqCMmmtx/iTYABbu0EEOhgvq/Ofx/IvPpzpAzAuT0tK34Y3JUlHm8nQKHTLOIiu+L+ rDbQ== X-Gm-Message-State: AOAM532kSWpPwMpnL8nftOUzF3oj4Ga5zxDNC7zvOaLCuBM1m3Q0W/E2 9On2twPPd9j7TE/qrcO4J5Iu4/roPgMLqkpDD8mTPPml X-Google-Smtp-Source: ABdhPJwe+F3n4hyJHsEjtBf0cNO3irFIPVr1dPjN8G67x/HisH7hOL0laadWJxt0Gk9loyM9tOd2H1JOevcMuR+PXCQ= X-Received: by 2002:a05:6830:13c7:: with SMTP id e7mr1558197otq.302.1641858187272; Mon, 10 Jan 2022 15:43:07 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:7f0a:0:b0:3f5:9e34:628e with HTTP; Mon, 10 Jan 2022 15:43:06 -0800 (PST) In-Reply-To: References: <20220110221116.gustgfgfge6pb5fe@mutt-hbsd> From: Mateusz Guzik Date: Tue, 11 Jan 2022 00:43:06 +0100 Message-ID: Subject: Re: Debugging a (potentially?) ZFS-related panic, and discussion about large patchsets To: Mark Johnston Cc: Shawn Webb , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4JXr3v6Qkpz3G76 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 1/11/22, Mark Johnston wrote: > On Mon, Jan 10, 2022 at 05:11:16PM -0500, Shawn Webb wrote: >> Hey all, >> >> So I'm getting an interesting ZFS-related kernel panic. I've uploaded >> the core.txt at [0]. I suspect it's related to FreeBSD commit >> 681ce946f33e75c590e97c53076e86dff1fe8f4a (zfs: merge >> openzfs/zfs@f291fa658 (master) into main). >> >> I'm able to reproduce it on a single system with some level of >> determinism: I'm building the security appliance firmware at ${DAYJOB} >> in a bhyve VM that's backed by a zvol. The host is a Dell Precision >> 7540 laptop with a single NVMe drive in it. The VM is configured with >> a single zvol, booting with UEFI. >> >> Looking at the commit email sent to dev-commits-src-all@, I see this: >> 146 files changed, 4933 insertions(+), 1572 deletions(-) >> >> Strangely, when I run `git show >> 681ce946f33e75c590e97c53076e86dff1fe8f4a`, I only see a small subset >> of those changes. > > That is a merge commit. You need to specify that you want a diff > against the first parent (the preceding FreeBSD), so something > equivalent to "git diff --stat 681ce946f^ 681ce946f". Use > "git log 681ce946f^2" to see the merged OpenZFS commits. > >> As a downstream consumer of 14-CURRENT, how am I supposed to even >> start debugging such a large patchset in any manner that respects my >> time? >> >> It seems to me that breaking up commits into smaller, bite-size chunks >> would make life easier for those experiencing bugs, especially ones >> that result in kernel panics. > > That's up to the upstream project, in this case OpenZFS. > >> ZFS in and of itself is a beast, and I've yet to study any of its >> code, so when there's a commit that large, even thinking about >> debugging it is a daunting task. >> >> Needless to say, I'm going to need some hand holding here for >> debugging this. Anyone have any idea what's going on? > > To start, you'll need to look at the stack trace for the thread with tid > 100061. > imo the kernel should be patched to obtain the trace on its own. As the target has interrupts disabled it will have to do it with NMI, but support for that got scrapped in commit 1c29da02798d968eb874b86221333a56393a94c3 Author: Mark Johnston Date: Fri Jan 31 15:43:33 2020 +0000 Reimplement stack capture of running threads on i386 and amd64. >> I guess this email is to serve three purposes: >> >> 1. Report that a bug was introduced recently. >> 2. Ask for help in squashing the bug. I'm more than happy to test any >> patches. >> 3. Start a dialogue on making life just a little easier for >> downstreams. >> >> [0]: https://hardenedbsd.org/~shawn/2022-01-10_zfs_core-r01.txt > > -- Mateusz Guzik