From nobody Mon Jan 10 22:11:16 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 25B3C1937EF0 for ; Mon, 10 Jan 2022 22:11:25 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JXp2428Hkz4h4D for ; Mon, 10 Jan 2022 22:11:24 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qt1-x82c.google.com with SMTP id f17so14059447qtf.8 for ; Mon, 10 Jan 2022 14:11:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:subject:message-id:mime-version:content-disposition; bh=pjSjuVC6BdsEVNT++l/IJSTWIlkXLhGI4uBoMBYzAOQ=; b=ETt+qlninjaki2gIO0OAPnU+5Kqxw1U2zHsQK4twoOet+0oHAN/F8ZImJc2+eWAOR3 6oJFNdF1VyUWT/DB7gklZXTnas2tfLxWdywZtOaL+JQ4y/EHCI82d8vVOsLt7E2yCF8N n8FEpMj/sDOsuz8O75pOHXan33NsQJtsb6wxqNDMSP25zq80+Ov+PLxho84TOwxV0at2 3mhVWHj86XrNyBxM1ZjOm/GaK6kSfkomIGs1ePGHZv5VfXFHJAG43zmxZt/InXLE/+1U T1OqGwgiR/f5q4p5sSdE+dZ0oXezS1tZMxEmj5XMIcEOG4ellTWvcteQXhASu11WvNuU O3Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition; bh=pjSjuVC6BdsEVNT++l/IJSTWIlkXLhGI4uBoMBYzAOQ=; b=ak9S8m4k/efzyVoE+GEbBtDhNNl+Sdk4cE3T1jHknP7iLMpxaFRS4x1Xs6uYyS3bNR XQIsh3u1JoaSJhvhvP2CsfV1dVcRa+BHvERESFVcHD7gFWfr8ufegXXDQduFFO7dvSi8 oKHzQYqCgrCcOrYzalO/vjq4VcLXubw2g9skNUkOd11jKSe3SkH69oCmIAqWnnlEdbUR /sGr20sdPXtaBkXuzdufXeZPBK4GENUzqrWx4lQrIlgY57gfY6D7xvarEMh70NnGdYPI 3SFHstXMc7EY/uI8E9kFLSyzDNaArZUnshH2A6eYRto78Coe4gsVS0m4BXzW1I7OVumi JIiw== X-Gm-Message-State: AOAM533+Xs983B4zzbu/7FllrncDfeVoLIxqRnWf4tIinBVcBV9ncSCO 9YZL2Qr5S6d2oBtscKi/4S1WjKUykJkISqGwGyFdSkE5r2DR1IKjCDi4AiWaKrIhkmg2lpeVv6S 2EI5D7n9g+Y+UWu3ayskvUyuedF41v+gSvimOl0HuMzaj4Mlszw3YJRsYMpSYWr/id2Gg978IeQ xKNqYtmoBXYQGt X-Google-Smtp-Source: ABdhPJxaYf/xXJ4gHAqBqLLMME3OsFdvbns8305Qo2npqsZKLgJdG8X4duyZQHVdNHnqlUs2wW3pOw== X-Received: by 2002:a05:622a:49:: with SMTP id y9mr1583407qtw.647.1641852677853; Mon, 10 Jan 2022 14:11:17 -0800 (PST) Received: from mutt-hbsd (pool-100-16-224-136.bltmmd.fios.verizon.net. [100.16.224.136]) by smtp.gmail.com with ESMTPSA id v10sm4938295qkp.105.2022.01.10.14.11.17 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 14:11:17 -0800 (PST) Date: Mon, 10 Jan 2022 17:11:16 -0500 From: Shawn Webb To: freebsd-hackers@freebsd.org Subject: Debugging a (potentially?) ZFS-related panic, and discussion about large patchsets Message-ID: <20220110221116.gustgfgfge6pb5fe@mutt-hbsd> X-Operating-System: FreeBSD mutt-hbsd 14.0-CURRENT-HBSD FreeBSD 14.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="sgfvw27rfranexkx" Content-Disposition: inline X-Rspamd-Queue-Id: 4JXp2428Hkz4h4D X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=ETt+qlni; dmarc=none; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::82c as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-2.92 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[hardenedbsd.org]; NEURAL_SPAM_SHORT(0.10)[0.102]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; NEURAL_SPAM_LONG(0.07)[0.074]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::82c:from]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_ALL(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[100.16.224.136:received] X-ThisMailContainsUnwantedMimeParts: N --sgfvw27rfranexkx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hey all, So I'm getting an interesting ZFS-related kernel panic. I've uploaded the core.txt at [0]. I suspect it's related to FreeBSD commit 681ce946f33e75c590e97c53076e86dff1fe8f4a (zfs: merge openzfs/zfs@f291fa658 (master) into main). I'm able to reproduce it on a single system with some level of determinism: I'm building the security appliance firmware at ${DAYJOB} in a bhyve VM that's backed by a zvol. The host is a Dell Precision 7540 laptop with a single NVMe drive in it. The VM is configured with a single zvol, booting with UEFI. Looking at the commit email sent to dev-commits-src-all@, I see this: 146 files changed, 4933 insertions(+), 1572 deletions(-) Strangely, when I run `git show 681ce946f33e75c590e97c53076e86dff1fe8f4a`, I only see a small subset of those changes. As a downstream consumer of 14-CURRENT, how am I supposed to even start debugging such a large patchset in any manner that respects my time? It seems to me that breaking up commits into smaller, bite-size chunks would make life easier for those experiencing bugs, especially ones that result in kernel panics. ZFS in and of itself is a beast, and I've yet to study any of its code, so when there's a commit that large, even thinking about debugging it is a daunting task. Needless to say, I'm going to need some hand holding here for debugging this. Anyone have any idea what's going on? I guess this email is to serve three purposes: 1. Report that a bug was introduced recently. 2. Ask for help in squashing the bug. I'm more than happy to test any patches. 3. Start a dialogue on making life just a little easier for downstreams. [0]: https://hardenedbsd.org/~shawn/2022-01-10_zfs_core-r01.txt Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --sgfvw27rfranexkx Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmHcrwIACgkQ/y5nonf4 4fqRHQ//abZ9zuVCCr/OSL5+mLZE9A+AZaTq21RIIFvDIhdSmCK01+zCAHMvuCps 6n1mtuYrpvgBY9Sa9aDc/zuseNPTJ604Y2jQADL30kMlTj8+qUAyORMF+/GtOlXi aQzcckSTQgxq/gnY0z6tDgcFtHs4F66IyTXEmRyDt15JX7rd8pO8Z/YyTgj61E1L Kuq88AjnBC5qmqB6AqqkPYm5piEZ2ZJq7JKPhVg6LDqi6wBEgY2zM1fPf3fLN1Dx YRNZnkI/xvoqQFrzFSqoTDRBI6Hj4OPJqAN6AmNCnEomy7HZH8HQZSJLDjyUKJrf /B6Bdo5arJGWBM+ov1WsRBjXVViGj0Fyy6JeVjzoZU439to9lehtZ0uVZf64EuH8 aNdFgcWXloTPOPLHnE623KbieJwQfFVgTsN3u0GeR1RxFN+bTMdV/Y58T3vPasAN CA1N2o3qONqa/MX/UUqQiyBEXE71QAzEzgXUaCnOLQZgdy7FDjz7pL5tyMta2Stz Rba4eFTcliqptKVrSKSHHkQUwl1iQSCOv8nyktiq6qwu17FzmbWnm0Cb+L1lzMAO UwygJH1gipwLMVszB8mE9vCjWz3N1EHuD06EMP/f4japEThfE7KasDRrFOZHFQVB 1cVr+uSx5Fori7w1gYQdOF1hkYpRajJOvX31+M3xA54mpTGvvHo= =EUgs -----END PGP SIGNATURE----- --sgfvw27rfranexkx--