From nobody Wed Jan 25 10:10:11 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P206N5DTYz3b43x for ; Wed, 25 Jan 2023 10:13:32 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay01.ispgateway.de (smtprelay01.ispgateway.de [80.67.18.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P206M4DgXz3qxs for ; Wed, 25 Jan 2023 10:13:31 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of freebsd-listen@fabiankeil.de has no SPF policy when checking 80.67.18.13) smtp.mailfrom=freebsd-listen@fabiankeil.de; dmarc=none Received: from [91.20.76.172] (helo=fabiankeil.de) by smtprelay01.ispgateway.de with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pKcoy-00063y-W7 for freebsd-hackers@freebsd.org; Wed, 25 Jan 2023 11:16:01 +0100 Date: Wed, 25 Jan 2023 11:10:11 +0100 From: Fabian Keil To: freebsd-hackers@freebsd.org Subject: Re: ZFS-related panic(s) with zfs-2.1.7-FreeBSD_g21bd76613? Message-ID: <20230125111011.455923bf@fabiankeil.de> In-Reply-To: <20230107174159.1b7e61e9@fabiankeil.de> References: <20230107174159.1b7e61e9@fabiankeil.de> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/cAAKNGfDZAS=R+BVPNb=P2T"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Df-Sender: Nzc1MDY3 X-Spamd-Result: default: False [-3.27 / 15.00]; SIGNED_PGP(-2.00)[]; AUTH_NA(1.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_SHORT(-0.97)[-0.971]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[80.67.18.13:from]; RCVD_IN_DNSWL_NONE(0.00)[80.67.18.13:from]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:34011, ipnet:80.67.16.0/20, country:DE]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[fabiankeil.de]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4P206M4DgXz3qxs X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --Sig_/cAAKNGfDZAS=R+BVPNb=P2T Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Fabian Keil wrote on 2023-01-07 at 17:41:59: > Yesterday I rebased ElectroBSD [0] on stable/13 77c0992af4e3b > while it was previously based on stable/13 d3b97a1ea0123. >=20 > I didn't notice any issues in a test VM and therefore decided > to update my laptop as well. >=20 > So far I've experienced three panics/reboots/freezes that I suspect > might be caused by the upgrade from zfs-2.1.6-FreeBSD_g6a6bd4939 > to zfs-2.1.7-FreeBSD_g21bd76613. >=20 > They all occurred while I was syncing ZFS datasets with zogftw [0]. >=20 > Unfortunately I only have one backtrace so I can't say for > sure that the other times where ZFS related as well: >=20 > Unread portion of the kernel message buffer: > panic: VERIFY3(0 =3D=3D zap_remove(mos, dsobj, spa_feature_table[f].fi_gu= id, tx)) failed (0 =3D=3D 2) >=20 > cpuid =3D 3 > time =3D 1673033419 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00dc6= 868a0 > vpanic() at vpanic+0x151/frame 0xfffffe00dc6868f0 > spl_panic() at spl_panic+0x3a/frame 0xfffffe00dc686950 > dsl_dataset_deactivate_feature_impl() at dsl_dataset_deactivate_feature_i= mpl+0xe6/frame 0xfffffe00dc6869a0 > dsl_dataset_clone_swap_sync_impl() at dsl_dataset_clone_swap_sync_impl+0x= 135/frame 0xfffffe00dc686ad0 > dmu_recv_end_sync() at dmu_recv_end_sync+0x2a2/frame 0xfffffe00dc686b30 > dsl_sync_task_sync() at dsl_sync_task_sync+0xb4/frame 0xfffffe00dc686b60 > dsl_pool_sync() at dsl_pool_sync+0x42b/frame 0xfffffe00dc686be0 > spa_sync() at spa_sync+0xb00/frame 0xfffffe00dc686e10 > txg_sync_thread() at txg_sync_thread+0x281/frame 0xfffffe00dc686ef0 > fork_exit() at fork_exit+0x7e/frame 0xfffffe00dc686f30 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00dc686f30 > --- trap 0x3, rip =3D 0xffffffff80659b3f, rsp =3D 0, rbp =3D 0xffffffff81= 8f4fa0 --- > mi_startup() at mi_startup+0xdf/frame 0xffffffff818f4fa0 > swapper() at swapper+0x69/frame 0xffffffff818f4ff0 > btext() at btext+0x22 > Uptime: 6m35s > Dumping 1098 out of 8050 MB:..2%..11%..21%..31%..41%..51%..62%..72%..81%.= .91% >=20 > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(struct pcpu, > (kgdb) where > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > #1 dump_savectx () at /usr/src/sys/kern/kern_shutdown.c:394 > #2 0xffffffff806cda18 in dumpsys (di=3D0x0) at /usr/src/sys/x86/include/= dump.h:87 > #3 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:423 > #4 kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:497 > #5 0xffffffff806cde9e in vpanic (fmt=3D, ap=3Dap@entry=3D= 0xfffffe00dc686930) at /usr/src/sys/kern/kern_shutdown.c:930 > #6 0xffffffff81278e3a in spl_panic (file=3D, func=3D, line=3D, fmt=3D) at /usr/src/sys/cont= rib/openzfs/module/os/freebsd/spl/spl_misc.c:107 > #7 0xffffffff813001e6 in dsl_dataset_deactivate_feature_impl (ds=3Dds@en= try=3D0xfffff80019a60000, f=3Df@entry=3DSPA_FEATURE_USEROBJ_ACCOUNTING, tx= =3Dtx@entry=3D0xfffff80191ace200) > at /usr/src/sys/contrib/openzfs/module/zfs/dsl_dataset.c:1116 > #8 0xffffffff81304cb5 in dsl_dataset_clone_swap_sync_impl (clone=3D0xfff= ff8018fc79000, origin_head=3D, tx=3D, tx@entry=3D= 0xfffff80191ace200) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_dataset.= c:4083 > #9 0xffffffff812eaff2 in dmu_recv_end_sync (arg=3D0xfffffe00d91366b8, tx= =3D0xfffff80191ace200) at /usr/src/sys/contrib/openzfs/module/zfs/dmu_recv.= c:3233 > #10 0xffffffff8132c254 in dsl_sync_task_sync (dst=3D0xfffffe00d91364a8, t= x=3Dtx@entry=3D0xfffff80191ace200) at /usr/src/sys/contrib/openzfs/module/z= fs/dsl_synctask.c:248 > #11 0xffffffff8131ea6b in dsl_pool_sync (dp=3Ddp@entry=3D0xfffff801eabad8= 00, txg=3Dtxg@entry=3D3576757) at /usr/src/sys/contrib/openzfs/module/zfs/d= sl_pool.c:847 > #12 0xffffffff81353930 in spa_sync_iterate_to_convergence (spa=3D0xfffffe= 00da149000, tx=3D0xfffff80191e73400) at /usr/src/sys/contrib/openzfs/module= /zfs/spa.c:9069 > #13 spa_sync (spa=3Dspa@entry=3D0xfffffe00da149000, txg=3Dtxg@entry=3D357= 6757) at /usr/src/sys/contrib/openzfs/module/zfs/spa.c:9287 > #14 0xffffffff81368281 in txg_sync_thread (arg=3Darg@entry=3D0xfffff801ea= bad800) at /usr/src/sys/contrib/openzfs/module/zfs/txg.c:591 > #15 0xffffffff80689fde in fork_exit (callout=3D0xffffffff81368000 , arg=3D0xfffff801eabad800, frame=3D0xfffffe00dc686f40) at /usr/s= rc/sys/kern/kern_fork.c:1093 > #16 > #17 mi_startup () at /usr/src/sys/kern/init_main.c:322 > #18 0xffffffff80a1e439 in swapper () at /usr/src/sys/vm/vm_swapout.c:755 > #19 0xffffffff802f8722 in btext () at /usr/src/sys/amd64/amd64/locore.S:80 >=20 > Has anyone else seen this? >=20 > I've seen it with three different ZFS pools and I think the pools > are fine. The laptop only supports USB2 so scrubbing the pools takes > days which is why I didn't do it yet. >=20 > I have no reliable way to reproduce the issue yet. > Running zogftw sync again after rebooting worked in all three cases. [...] > [0] > [1] I'm still getting panics with the stack trace above with various pools about once a day on my work laptop and am considering reverting the ZFS-related commits. Fabian --Sig_/cAAKNGfDZAS=R+BVPNb=P2T Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQTKUNd6H/m3+ByGULIFiohV/3dUnQUCY9EAAwAKCRAFiohV/3dU ne9/AKDOcNMKAuRrARnAIE3Hwujq3eFEJQCeM8ar+U5rljnSh8PqJVL4SjEzfhw= =kTZC -----END PGP SIGNATURE----- --Sig_/cAAKNGfDZAS=R+BVPNb=P2T--