From nobody Wed Aug 13 10:37:45 2025 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4c24Zh004Kz64YHZ for ; Wed, 13 Aug 2025 10:37:47 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4c24Zg5gz1z41FM; Wed, 13 Aug 2025 10:37:47 +0000 (UTC) (envelope-from dim@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1755081467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8lt/TAbuGQB63dhZZ2Ti90LsripU5e+avPYxre0BUfs=; b=D8l5i7+BfA/MjV01EQFg+MHVcF41gTunAaYEQIxBwC2+opAs4kkVf103aR0blY5tlWn/C/ ZXlASNswgYR4ukD/SA8TlPt3vuy1bCfODnzPu+PmtjyMTRko5lDNltwp/IgHnHwJtbWZOy O4x91tz/SVBhEdAftbLVzxHLGzabe0C3Ll8Hk/4ONOj0jYp+W94H2nqJr9DGoC9hfa4LHP QlswtXy1DNs/iRpj9iQ7rkhp0daEdiEDkK7D1LQKMtC4Ibyq7dgnlZHRx+0cfybViEP7uH JtLI/72TJiHpLqdtXbfFA2W2s6EqRh464c3DmRaCePPEjuQymNizbOM2QsW7Kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1755081467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8lt/TAbuGQB63dhZZ2Ti90LsripU5e+avPYxre0BUfs=; b=XOAyI5jXPWvmuyJdOepQzGpQIvE4npsSXVaaaFE5dUiNeGyMIx7oCEOXYU1oUcaYllSB4c a5dIspEBRIHEAA7VQGCwGhabwyUFXWtfHS7IIZD1ALRrUGql9TMz1OOknzPSpnzKMjbO5U PgX/sNyNHXdGOLP8ijLPtSi/m/1pswRX1LiDlJEmJfFxVz8OrdbCYXpISUym9ACfrVcNAg ooy4w0QxGpWHQ1kGiwEuq8MTWZ/6ACNd8kNSHgXGn/rzmoQoI9oGc0qBxMz2Ds6KuvtSEZ qO94kQqC5IV88rGe9m96DbQoHFPtGUsHpSBsOrbdBhLniz4n1cqs/GMFbRB5ag== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1755081467; a=rsa-sha256; cv=none; b=hAvj62eAwtHQxoB8MFayAnOj5FAlOhrZTrHzMwL0IVRfEPD05xN0CE7OjmsHENlBWFv4XC SaeHFQMWi1oDJLZYDJRUotNyWL5WaXT7PG4Ad2e6xhmlEEOHVe8D7qFhBCKVVwpf+WYXOx HTlbb0sFh5K7nylOTEWtzlNyhY/vcWKvyS02l6EOeQnktUWDEJumeVTw6GQ4XELwAcktK+ hXPFO8weN3tHCvI97pevBf1JFTlI5+SjrAyrUwVcg3FTy9/+KXx5SbtM2q7adJx2FBuG/Y yEF7QwrKgqs8Y/WktZL3mUHNmI5tFoAeQ96M4j8aSDdep98LfYwGdqO3V30odQ== Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "tensor.andric.com", Issuer "R11" (verified OK)) (Authenticated sender: dim) by smtp.freebsd.org (Postfix) with ESMTPSA id 4c24Zg4K0zz16SM; Wed, 13 Aug 2025 10:37:47 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtpclient.apple (bladnoch.home.andric.com [192.168.0.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 0F7956C25E; Wed, 13 Aug 2025 12:37:46 +0200 (CEST) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: panic in ZFS: VERIFY3U(BP_GET_BIRTH(bp), ==, BP_GET_BIRTH(&found->le_bp)) failed (15427394 == 15427393) From: Dimitry Andric In-Reply-To: <3b14032a60cd99670c5619d2818f549f@Leidinger.net> Date: Wed, 13 Aug 2025 12:37:45 +0200 Cc: Current Content-Transfer-Encoding: quoted-printable Message-Id: References: <3b14032a60cd99670c5619d2818f549f@Leidinger.net> To: Alexander Leidinger X-Mailer: Apple Mail (2.3826.700.81) On 13 Aug 2025, at 11:46, Alexander Leidinger = wrote: >=20 > I get this panic with -current as of 2025-08-11-154054 (CEST): > ---snip--- > (kgdb) #0 __curthread () at = /space/system/usr_src/sys/amd64/include/pcpu_aux.h:57 > td =3D > #1 doadump (textdump=3Dtextdump@entry=3D1) > at /space/system/usr_src/sys/kern/kern_shutdown.c:399 > error =3D 0 > coredump =3D > #2 0xffffffff8053b305 in kern_reboot (howto=3D260) > at /space/system/usr_src/sys/kern/kern_shutdown.c:519 > once =3D 0 > __pc =3D 0x0 > #3 0xffffffff8053b837 in vpanic ( > fmt=3D0xffffffff816d11f0 "VERIFY3U(BP_GET_BIRTH(bp), =3D=3D, = BP_GET_BIRTH(&found->le_bp)) failed (%llu =3D=3D %llu)\n", = ap=3Dap@entry=3D0xfffffe0420634b30) > at /space/system/usr_src/sys/kern/kern_shutdown.c:974 > buf =3D "VERIFY3U(BP_GET_BIRTH(bp), =3D=3D, = BP_GET_BIRTH(&found->le_bp)) failed (15427394 =3D=3D 15427393)\n", = '\000' > __pc =3D 0x0 > __pc =3D 0x0 > __pc =3D 0x0 > other_cpus =3D {__bits =3D {16777213, 0 }} > td =3D 0xfffff801228ee780 > bootopt =3D > newpanic =3D > #4 0xffffffff8140650a in spl_panic (file=3D, > func=3D, line=3D, fmt=3D) > at = /space/system/usr_src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c= :100 > ap =3D {{gp_offset =3D 48, fp_offset =3D 48, > overflow_arg_area =3D 0xfffffe0420634b60, > reg_save_area =3D 0xfffffe0420634b00}} > #5 0xffffffff814d4615 in dsl_livelist_iterate (arg=3D, > bp=3D, bp_freed=3D0, tx=3D) > at = /space/system/usr_src/sys/contrib/openzfs/module/zfs/dsl_deadlist.c:1052 > _verify3_left =3D > _verify3_right =3D > node =3D {le_bp =3D {blk_dva =3D {{dva_word =3D {24, = 5140063200}}, { > dva_word =3D {0, 0}}, {dva_word =3D {0, 0}}}, > blk_prop =3D 9228727766583607296, blk_prop2 =3D 0, blk_pad = =3D 0, > blk_birth_word =3D {15427393, 15427394}, blk_fill =3D 0, = blk_cksum =3D { > zc_word =3D {0, 0, 0, 0}}}, le_refcnt =3D 9, le_node =3D = {avl_child =3D { > 0xfffffe0420634c30, 0xffffffff81492e2a = }, > avl_pcb =3D 18446735324294374176}} > lia =3D > avl =3D 0xfffffe0420634d88 > to_free =3D > t =3D > found =3D > #6 0xffffffff8146a402 in bpobj_iterate_blkptrs ( > bpi=3Dbpi@entry=3D0xfffff80eca95e3c0, > func=3Dfunc@entry=3D0xffffffff814d4320 , > arg=3Darg@entry=3D0xfffffe0420634da8, start=3Dstart@entry=3D0, = tx=3Dtx@entry=3D0x0, > free=3Dfree@entry=3D0) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/bpobj.c:325 > bp =3D 0xfffffe07b50a4880 > blkoff =3D > bparray =3D > offset =3D 6272 > bp_freed =3D 0 > dbuf =3D 0xfffff806386e5bc0 > err =3D > freed =3D 0 > comp_freed =3D 0 > uncomp_freed =3D 0 > bpo =3D 0xfffff80ade376320 > i =3D 49 > pe =3D > ps =3D 0 > pb =3D 0 > #7 0xffffffff814699c1 in bpobj_iterate_impl ( > initial_bpo=3Dinitial_bpo@entry=3D0xfffff80ade376320, > func=3D0xffffffff814d4320 , > arg=3Darg@entry=3D0xfffffe0420634da8, tx=3Dtx@entry=3D0x0, = free=3Dfree@entry=3D0, > bpobj_size=3Dbpobj_size@entry=3D0x0) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/bpobj.c:401 > bpo =3D 0xfffff80ade376320 > stack =3D {list_offset =3D 40, list_head =3D { > list_next =3D 0xfffff80eca95e3e8, list_prev =3D = 0xfffff80eca95e3e8}} > err =3D > bpi =3D 0xfffff80eca95e3c0 > #8 0xffffffff8146a0d2 in bpobj_iterate_nofree (bpo=3D, > bpo@entry=3D0xfffff80ade376320, func=3D, = arg=3D, > arg@entry=3D0xfffffe0420634da8, bpobj_size=3D, > bpobj_size@entry=3D0x0) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/bpobj.c:567 > No locals. > #9 0xffffffff814d422c in dsl_process_sub_livelist ( > bpobj=3Dbpobj@entry=3D0xfffff80ade376320, > to_free=3Dto_free@entry=3D0xfffffe0420634e08, = t=3Dt@entry=3D0xfffff803e479e300, > size=3Dsize@entry=3D0x0) > at = /space/system/usr_src/sys/contrib/openzfs/module/zfs/dsl_deadlist.c:1106 > avl =3D {avl_root =3D 0xfffff811914e9488, > avl_compar =3D 0xffffffff814d42e0 , > avl_offset =3D 136, avl_numnodes =3D 25} > arg =3D {avl =3D 0xfffffe0420634d88, to_free =3D = 0xfffffe0420634e08, > t =3D 0xfffff803e479e300} > cookie =3D 0xfffff80ade376300 > err =3D > le =3D > #10 0xffffffff81536a52 in spa_livelist_delete_cb = (arg=3D0xfffffe022835d000, > z=3D0xfffff803e479e300) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/spa.c:3223 > to_free =3D {bpl_lock =3D {lock_object =3D { > lo_name =3D 0xffffffff816f57e9 <.L.str+1> = "bpl->bpl_lock", > lo_flags =3D 577830912, lo_data =3D 0, lo_witness =3D = 0x0}, > sx_lock =3D 1}, bpl_list =3D {list_offset =3D 128, = list_head =3D { > list_next =3D 0xfffff80a5a71e580, list_prev =3D = 0xfffff8101a903680}}} > ll =3D 0xfffff8053f3e1000 > dle =3D 0xfffff80ade376300 > err =3D > count =3D 1 > spa =3D 0xfffffe022835d000 > ll_obj =3D > mos =3D > zap_obj =3D 14665 > #11 0xffffffff8164c16f in zthr_procedure (arg=3D0xfffff803e479e300) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/zthr.c:246 > t =3D 0xfffff803e479e300 > #12 0xffffffff804ee0f7 in fork_exit ( > callout=3D0xffffffff8164c0c0 , = arg=3D0xfffff803e479e300, > frame=3D0xfffffe0420634f40) > at /space/system/usr_src/sys/kern/kern_fork.c:1153 > __pc =3D 0x0 > __pc =3D 0x0 > td =3D 0xfffff801228ee780 > p =3D 0xfffffe021d808558 > dtd =3D >=20 > Tracing command "zfskern", '\000' pid 6 tid 101414 = (CPU 1) > #0 __curthread () at = /space/system/usr_src/sys/amd64/include/pcpu_aux.h:57 > #1 doadump (textdump=3Dtextdump@entry=3D1) > at /space/system/usr_src/sys/kern/kern_shutdown.c:399 > #2 0xffffffff8053b305 in kern_reboot (howto=3D260) > at /space/system/usr_src/sys/kern/kern_shutdown.c:519 > #3 0xffffffff8053b837 in vpanic ( > fmt=3D0xffffffff816d11f0 "VERIFY3U(BP_GET_BIRTH(bp), =3D=3D, = BP_GET_BIRTH(&found->le_bp)) failed (%llu =3D=3D %llu)\n", = ap=3Dap@entry=3D0xfffffe0420634b30) > at /space/system/usr_src/sys/kern/kern_shutdown.c:974 > #4 0xffffffff8140650a in spl_panic (file=3D, > func=3D, line=3D, fmt=3D) > at = /space/system/usr_src/sys/contrib/openzfs/module/os/freebsd/spl/spl_misc.c= :100 > #5 0xffffffff814d4615 in dsl_livelist_iterate (arg=3D, > bp=3D, bp_freed=3D0, tx=3D) > at = /space/system/usr_src/sys/contrib/openzfs/module/zfs/dsl_deadlist.c:1052 > #6 0xffffffff8146a402 in bpobj_iterate_blkptrs ( > bpi=3Dbpi@entry=3D0xfffff80eca95e3c0, > func=3Dfunc@entry=3D0xffffffff814d4320 , > arg=3Darg@entry=3D0xfffffe0420634da8, start=3Dstart@entry=3D0, = tx=3Dtx@entry=3D0x0, > free=3Dfree@entry=3D0) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/bpobj.c:325 > #7 0xffffffff814699c1 in bpobj_iterate_impl ( > initial_bpo=3Dinitial_bpo@entry=3D0xfffff80ade376320, > func=3D0xffffffff814d4320 , > arg=3Darg@entry=3D0xfffffe0420634da8, tx=3Dtx@entry=3D0x0, = free=3Dfree@entry=3D0, > bpobj_size=3Dbpobj_size@entry=3D0x0) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/bpobj.c:401 > #8 0xffffffff8146a0d2 in bpobj_iterate_nofree (bpo=3D, > bpo@entry=3D0xfffff80ade376320, func=3D, = arg=3D, > arg@entry=3D0xfffffe0420634da8, bpobj_size=3D, > bpobj_size@entry=3D0x0) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/bpobj.c:567 > #9 0xffffffff814d422c in dsl_process_sub_livelist ( > bpobj=3Dbpobj@entry=3D0xfffff80ade376320, > to_free=3Dto_free@entry=3D0xfffffe0420634e08, = t=3Dt@entry=3D0xfffff803e479e300, > size=3Dsize@entry=3D0x0) > at = /space/system/usr_src/sys/contrib/openzfs/module/zfs/dsl_deadlist.c:1106 > #10 0xffffffff81536a52 in spa_livelist_delete_cb = (arg=3D0xfffffe022835d000, > z=3D0xfffff803e479e300) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/spa.c:3223 > #11 0xffffffff8164c16f in zthr_procedure (arg=3D0xfffff803e479e300) > at /space/system/usr_src/sys/contrib/openzfs/module/zfs/zthr.c:246 > #12 0xffffffff804ee0f7 in fork_exit ( > callout=3D0xffffffff8164c0c0 , = arg=3D0xfffff803e479e300, > frame=3D0xfffffe0420634f40) > at /space/system/usr_src/sys/kern/kern_fork.c:1153 > #13 > ---snip--- This is due to = https://cgit.freebsd.org/src/commit/?id=3Ddf58e8b1506f241670be86a560fb6e84= 32043aee, see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D288722 = . It's currently unknown what the root cause is. We need some ZFS wizards. = :) -Dimitry