From nobody Wed Apr 12 13:16:07 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PxNWm5JYrz44lFq for ; Wed, 12 Apr 2023 13:16:20 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PxNWm1x6lz4JPN; Wed, 12 Apr 2023 13:16:20 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1681305380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jNWkNy575vt6sIkSkNLjkY1jU4qox3NeeGnLNQSik7g=; b=uectMiZqtT66oi6XXJCSOU8LEmEk+TRJw9qZ/owg/gX9Ov1gr5SUQdKT7Tr5wiHF6Gxza/ Y18AnIbPlsiH3OJ7sUStaJ9ksmfZ0BzF7gH1MTScR/ZHGRnpMAXNw4kZoVYZ/sGRd+QyhA fFak7BQqXrucjhlvYOwkhpvJgj2xbsR+8g0/wrL+bOfph4MZY7SnjQq1jBLHbT1f6e7Xee RnTGLDrxEb3xaBWOQXgx0jirD2i8ImqsF6HZhS1GSCOhgeDohl2c4fPZPTUzF4yEkJbp5p a/9uANsfnG+C1pOfCjR5nYkk8DP7Xa5ON4d12/Yj1YlDKsr7KLkWzKDIBW8Dvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1681305380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jNWkNy575vt6sIkSkNLjkY1jU4qox3NeeGnLNQSik7g=; b=MtMrQKX7evo7M5oiSYHm31ZfYcz57sLp4CsB4NZszY9O3zbamKQouCRCPKqeZhB9x/CRuq VZiQtnLxRSQkVKEikEVpHf8Ni8/tfaA9NTaRwfPAv7poBMA8EQf3iuDGg5yUlxv0bRrL9i vfCQIKSUlWvTX9fYNE/PzsAHPIJBtH+4FMnp7L28IH1727kxuY1Y6WEdTnTIg1PuaTAygI JbKtsNST5pD8k78gxZFfVoL+CnaTumEIBVEGdnXEi9PLhea9UJ7RYUjg5xwhiEPE4aOtdB QpXKKxUM0TGEJfodkv06ZBV3/wjOTWYODYFvcFgcdIA0Swk2o/kKVqKaYO9+oA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1681305380; a=rsa-sha256; cv=none; b=RGfZ5ORzCvH315WW8nygJH0G4fN53kQaXErtB8K4sbtP59+QbDJJ6xQzLKt7ZjK4dTImB3 42v5+lU4RCDkKcgGf8DVCVjBNVNB7UHvW2Cc/1U4iYcaLdccs1Hu47BTUx8l5rWfwmjde1 wvBSKVJ1bC/O8Kn7oUzPrFE2WTJNIjIFftKYgdijmoww35mNJ+iGwuZHtwqV7r2N+8Z0vU a9u2LUmaZxhJqwfptucDiMnorwXJSSqkxZL3VJ+ifx0yHokcujiZ1+fxBJjH+mgSIj25cF njAwB0C80xhGC9e7JWNT1hcSGdMMji92bve7CPLqUab/snhZMQJwkeNjV3o3pQ== Received: from mail-vs1-f53.google.com (mail-vs1-f53.google.com [209.85.217.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PxNWm0gbKz1S2f; Wed, 12 Apr 2023 13:16:20 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-vs1-f53.google.com with SMTP id v10so10328422vsf.6; Wed, 12 Apr 2023 06:16:20 -0700 (PDT) X-Gm-Message-State: AAQBX9cHKPr+yqSvDv3gZNcLeI4CkyQ+BBnWnzIG0wx8Mex6H1gZouv0 2G/FlN2Ogsw+phWnh47lMLvr5jdUsxzkAz0gsAY= X-Google-Smtp-Source: AKy350b/dynGJAPqI0FhXB/SzkU+SVyrYoRrJ3XpBqUTMPDs4TE7Vu2f26H09Q/zsjuVbyAKCX4hyHVnnc0tRAeoYuM= X-Received: by 2002:a67:d891:0:b0:42c:4297:81b0 with SMTP id f17-20020a67d891000000b0042c429781b0mr4114727vsj.7.1681305379537; Wed, 12 Apr 2023 06:16:19 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <20230411021919.0718F306@slippy.cwsent.com> <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> <20230411142831.DB8245FA@slippy.cwsent.com> <20230411144713.A94EA5FE@slippy.cwsent.com> In-Reply-To: From: Nuno Teixeira Date: Wed, 12 Apr 2023 14:16:07 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) To: Cy Schubert Cc: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= , FreeBSD User , Mateusz Guzik , Pawel Jakub Dawidek , FreeBSD CURRENT Content-Type: multipart/alternative; boundary="000000000000e80a0f05f9236911" X-ThisMailContainsUnwantedMimeParts: N --000000000000e80a0f05f9236911 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable (...) Trying : `zpool set feature@block_cloning=3Ddisabled zroot`: cannot set property for 'zroot': property 'feature@block_cloning' can only be set to 'disabled' at creation time Nuno Teixeira escreveu no dia quarta, 12/04/2023 =C3= =A0(s) 13:57: > Hello all, > > at current 3fdb40d1befe after `zfs upgrade XXX`: > > same problem when running compiler: > > - poudriere: crash without dump > - make buildworld (/usr/src): shutdown -p (I will try to get a photo) > > Is there a way to disable block clone? > > Cy Schubert escreveu no dia ter=C3=A7a, 11/04= /2023 > =C3=A0(s) 15:47: > >> In message <20230411142831.DB8245FA@slippy.cwsent.com>, Cy Schubert >> writes: >> > In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net>, >> > =3D?utf-8?Q?Pawe=3DC >> > 5=3D82_Jakub_Dawidek?=3D writes: >> > > >> > > >> > > > On Apr 11, 2023, at 11:31, Cy Schubert >> wrote: >> > > >=3D20 >> > > > =3DEF=3DBB=3DBFIn message >> <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn. >> > d=3D >> > > e>,=3D20 >> > > > FreeBSD Us >> > > > er writes: >> > > >> Am Sun, 9 Apr 2023 14:37:03 +0200 >> > > >> Mateusz Guzik schrieb: >> > > >>=3D20 >> > > >>>> On 4/9/23, FreeBSD User wrote: >> > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 >> main-n262052-0d4038 >> > e=3D >> > > 301 >> > > >>> 2b: >> > > >>>>> Sun Apr 9 >> > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via >> > > >>>>>=3D20 >> > > >>>>> zpool upgrade POOLNAME >> > > >>>>>=3D20 >> > > >>>>> some boxes keep crashing when starting compiler runs (the >> trigger is >> > > >>>>> different on boxes). >> > > >>>>>=3D20 >> > > >>>>> ZFS module is statically compiled into the kernel (if this is = of >> > > >>>>> importance) >> > > >>>>>=3D20 >> > > >>>>> Last known good was: >> > > >>>>>=3D20 >> > > >>>>> [...] >> > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 >> > > >>>>> main-n262051-75379ea2e461: Sun Apr >> > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: >> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 >> 07:10:04 >> > < >> > > =3D >> > > 0. >> > > >>> 2> >> > > >>>>> thor kernel: >> > > >>>>> FreeBSD clang version 15.0.7 ( >> https://github.com/llvm/llvm-project.gi >> > t=3D >> > > >> > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor >> kernel: >> > > >>>>> VT(efifb): resolution >> > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl >> already >> > > >>>>> present! >> > > >>>>> [...] >> > > >>>>>=3D20 >> > > >>>>> The file /var/crash/info.X >> > > >>>>>=3D20 >> > > >>>>> contains: >> > > >>>>>=3D20 >> > > >>>>> [...] >> > > >>>>>=3D20 >> > > >>>>> root@thor:/var/crash # more info.2 >> > > >>>>> Dump header from device: /dev/gpt/swap >> > > >>>>> Architecture: amd64 >> > > >>>>> Architecture Version: 2 >> > > >>>>> Dump Length: 1095192576 >> > > >>>>> Blocksize: 512 >> > > >>>>> Compression: none >> > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000 >> > > >>>>> Hostname: thor.local >> > > >>>>> Magic: FreeBSD Kernel Dump >> > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 >> main-n262052-0d4038e3012b: S >> > u=3D >> > > n=3D20 >> > > >>> Apr >> > > >>>>> 9 12:01:02 CEST >> > > >>>>> 2023 >> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR >> > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed >> > > >>>>>=3D20 >> > > >>>>> Dump Parity: 2961465682 >> > > >>>>> Bounds: 2 >> > > >>>>> Dump Status: good >> > > >>>>>=3D20 >> > > >>>>> Until reconfigured for more debug stuff I do not have more to >> present >> > .=3D >> > > >> > > >>>>>=3D20 >> > > >>>>> I rememeber now really scraed that there was a HEADSUP in the >> list re >> > g=3D >> > > ard >> > > >>> ing >> > > >>>>> some serious ZFS >> > > >>>>> problems - I didn't find it right now. >> > > >>>>>=3D20 >> > > >>>>> Thanks in advance, >> > > >>>>>=3D20 >> > > >>>=3D20 >> > > >>> That's fallout from the new block cloning feature, adding the >> author >> > > >>>=3D20 >> > > >>=3D20 >> > > >> Thanks. >> > > >>=3D20 >> > > >> As of this moment, all systems with the newest kernel and the new >> ZFS op >> > t=3D >> > > ion=3D20 >> > > >> enabled, crash - >> > > >> the reason is mostly in different ZFS datasets. I guess there is >> no way >> > b >> > > =3D >> > > ack >> > > >> once this faulty >> > > >> option is enabled? >> > > >=3D20 >> > > > I've run a test on a scratch pool here, first without >> block_cloning=3D20 >> > > > enabled, then with. There was no corruption when block_cloning >> was=3D20 >> > > > disabled. There was corruption when block_cloning was enabled. >> > > >=3D20 >> > > > I don't know of any way to revert back nor is there any way to fix >> or=3D20 >> > > > recover the corrupted blocks. >> > > >> > > Is the corruption still present after EXDEV fixes? >> > >> > Yes and no. >> > >> > Yes, there is corruption when block_cloning is enabled. >> > >> > There is no corruption when block_cloning is disabled. >> >> I should add some detail to this. >> >> The corruption experienced when block cloning is disabled was fixed by: >> >> - eb1feadc201a >> - e2d997d1cbb9 >> - d012836fb616 (specifically this commit) >> - 20be1b4fc4b7 >> >> When block_cloning is enabled, the pool is corrupted. This has not been >> fixed. >> >> >> -- >> Cheers, >> Cy Schubert >> FreeBSD UNIX: Web: https://FreeBSD.org >> NTP: Web: https://nwtime.org >> >> e^(i*pi)+1=3D0 >> >> >> >> > > -- > Nuno Teixeira > FreeBSD Committer (ports) > --=20 Nuno Teixeira FreeBSD Committer (ports) --000000000000e80a0f05f9236911 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
(...)

Trying :
`zpool set feature@block_cloning=3Ddisabled zroot`:
cannot s= et property for 'zroot': property 'feature@block_cloning' c= an only be set to 'disabled' at creation time

Nuno Teixeira &l= t;eduardo@freebsd.org> escrev= eu no dia quarta, 12/04/2023 =C3=A0(s) 13:57:
Hello all,
at current 3fdb40d1befe after `zfs upgrade XXX`:

same problem when running compiler:

- poudriere: crash without dump
- make buildworld (/usr/src): s= hutdown -p (I will try to get a photo)

Is there a = way to disable block clone?

<= div dir=3D"ltr" class=3D"gmail_attr">Cy Schubert <Cy.Schubert@cschubert.com> = escreveu no dia ter=C3=A7a, 11/04/2023 =C3=A0(s) 15:47:
In message <20230411142831.D= B8245FA@slippy.cwsent.com>, Cy Schubert writes:
> In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek= .net>,
> =3D?utf-8?Q?Pawe=3DC
> 5=3D82_Jakub_Dawidek?=3D writes:
> >
> >
> > > On Apr 11, 2023, at 11:31, Cy Schubert <Cy.Schubert@cschubert.com= > wrote:
> > >=3D20
> > > =3DEF=3DBB=3DBFIn message <20230409161436.5412fa6e@thor.i= ntern.walstatt.dynvpn.
> d=3D
> > e>,=3D20
> > > FreeBSD Us
> > > er writes:
> > >> Am Sun, 9 Apr 2023 14:37:03 +0200
> > >> Mateusz Guzik <mjguzik@gmail.com> schrieb:
> > >>=3D20
> > >>>> On 4/9/23, FreeBSD User <freebsd@walstatt-de.de> wrot= e:
> > >>>>> Today, after upgrading to FreeBSD 14.0-CURRE= NT #8 main-n262052-0d4038
> e=3D
> > 301
> > >>> 2b:
> > >>>>> Sun Apr=C2=A0 9
> > >>>>> 12:01:02 CEST 2023=C2=A0 amd64, AND upgradin= g ZPOOLs via
> > >>>>>=3D20
> > >>>>> zpool upgrade POOLNAME
> > >>>>>=3D20
> > >>>>> some boxes keep crashing when starting compi= ler runs (the trigger is
> > >>>>> different on boxes).
> > >>>>>=3D20
> > >>>>> ZFS module is statically compiled into the k= ernel (if this is of
> > >>>>> importance)
> > >>>>>=3D20
> > >>>>> Last known good was:
> > >>>>>=3D20
> > >>>>> [...]
> > >>>>> Apr=C2=A0 9 07:10:04 <0.2> thor kernel= : FreeBSD 14.0-CURRENT #7
> > >>>>> main-n262051-75379ea2e461: Sun Apr
> > >>>>> 9 00:12:57 CEST 2023 Apr=C2=A0 9 07:10:04 &l= t;0.2> thor kernel:
> > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/T= HOR amd64 Apr=C2=A0 9 07:10:04
>=C2=A0 <
> > =3D
> > 0.
> > >>> 2>
> > >>>>> thor kernel:
> > >>>>> FreeBSD clang version 15.0.7 (= https://github.com/llvm/llvm-project.gi
> t=3D
> >
> > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr=C2=A0 9 = 07:10:04 <0.2> thor kernel:
> > >>>>> VT(efifb): resolution
> > >>>>> 2560x1440 Apr=C2=A0 9 07:10:04 <0.2> t= hor kernel: module zfsctrl already
> > >>>>> present!
> > >>>>> [...]
> > >>>>>=3D20
> > >>>>> The file /var/crash/info.X
> > >>>>>=3D20
> > >>>>> contains:
> > >>>>>=3D20
> > >>>>> [...]
> > >>>>>=3D20
> > >>>>> root@thor:/var/crash # more info.2
> > >>>>> Dump header from device: /dev/gpt/swap
> > >>>>>=C2=A0 Architecture: amd64
> > >>>>>=C2=A0 Architecture Version: 2
> > >>>>>=C2=A0 Dump Length: 1095192576
> > >>>>>=C2=A0 Blocksize: 512
> > >>>>>=C2=A0 Compression: none
> > >>>>>=C2=A0 Dumptime: 2023-04-09 11:43:41 +0000 > > >>>>>=C2=A0 Hostname: thor.local
> > >>>>>=C2=A0 Magic: FreeBSD Kernel Dump
> > >>>>>=C2=A0 Version String: FreeBSD 14.0-CURRENT #= 8 main-n262052-0d4038e3012b: S
> u=3D
> > n=3D20
> > >>> Apr
> > >>>>> 9 12:01:02 CEST
> > >>>>> 2023
> > >>>>>=C2=A0 =C2=A0 root@thor:/usr/obj/usr/src/amd6= 4.amd64/sys/THOR
> > >>>>>=C2=A0 Panic String: VERIFY(!zil_replaying(zi= log, tx)) failed
> > >>>>>=3D20
> > >>>>>=C2=A0 Dump Parity: 2961465682
> > >>>>>=C2=A0 Bounds: 2
> > >>>>>=C2=A0 Dump Status: good
> > >>>>>=3D20
> > >>>>> Until reconfigured for more debug stuff I do= not have more to present
> .=3D
> >
> > >>>>>=3D20
> > >>>>> I rememeber now really scraed that there was= a HEADSUP in the list re
> g=3D
> > ard
> > >>> ing
> > >>>>> some serious ZFS
> > >>>>> problems - I didn't find it right now. > > >>>>>=3D20
> > >>>>> Thanks in advance,
> > >>>>>=3D20
> > >>>=3D20
> > >>> That's fallout from the new block cloning featur= e, adding the author
> > >>>=3D20
> > >>=3D20
> > >> Thanks.
> > >>=3D20
> > >> As of this moment, all systems with the newest kernel an= d the new ZFS op
> t=3D
> > ion=3D20
> > >> enabled, crash -
> > >> the reason is mostly in=C2=A0 different ZFS datasets. I = guess there is no way
>=C2=A0 b
> > =3D
> > ack
> > >> once this faulty
> > >> option is enabled?
> > >=3D20
> > > I've run a test on a scratch pool here, first without bl= ock_cloning=3D20
> > > enabled, then with. There was no corruption when block_cloni= ng was=3D20
> > > disabled. There was corruption when block_cloning was enable= d.
> > >=3D20
> > > I don't know of any way to revert back nor is there any = way to fix or=3D20
> > > recover the corrupted blocks.
> >
> > Is the corruption still present after EXDEV fixes?
>
> Yes and no.
>
> Yes, there is corruption when block_cloning is enabled.
>
> There is no corruption when block_cloning is disabled.

I should add some detail to this.

The corruption experienced when block cloning is disabled was fixed by:

- eb1feadc201a
- e2d997d1cbb9
- d012836fb616 (specifically this commit)
- 20be1b4fc4b7

When block_cloning is enabled, the pool is corrupted. This has not been fixed.


--
Cheers,
Cy Schubert <Cy.Schubert@cschubert.com>
FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 https://FreeB= SD.org
NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<cy@nwtime.org>=C2=A0 =C2=A0 Web:=C2=A0 https://nwt= ime.org

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 e^(i*pi)+1=3D0





--
Nuno TeixeiraFreeBSD Committer (ports)


--
Nuno Teixeira
FreeBSD Committ= er (ports)
--000000000000e80a0f05f9236911--