Date: Wed, 12 Apr 2023 13:57:02 +0100 From: Nuno Teixeira <eduardo@freebsd.org> To: Cy Schubert <Cy.Schubert@cschubert.com> Cc: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= <pawel@dawidek.net>, FreeBSD User <freebsd@walstatt-de.de>, Mateusz Guzik <mjguzik@gmail.com>, Pawel Jakub Dawidek <pjd@freebsd.org>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) Message-ID: <CAFDf7UL6d-EyEhcPGjZfEia_qO9M%2BxY4bQMsD9hemmHYapQU4Q@mail.gmail.com> In-Reply-To: <20230411144713.A94EA5FE@slippy.cwsent.com> References: <20230411021919.0718F306@slippy.cwsent.com> <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> <20230411142831.DB8245FA@slippy.cwsent.com> <20230411144713.A94EA5FE@slippy.cwsent.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Hello all, at current 3fdb40d1befe after `zfs upgrade XXX`: same problem when running compiler: - poudriere: crash without dump - make buildworld (/usr/src): shutdown -p (I will try to get a photo) Is there a way to disable block clone? Cy Schubert <Cy.Schubert@cschubert.com> escreveu no dia terça, 11/04/2023 à(s) 15:47: > In message <20230411142831.DB8245FA@slippy.cwsent.com>, Cy Schubert > writes: > > In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net>, > > =?utf-8?Q?Pawe=C > > 5=82_Jakub_Dawidek?= writes: > > > > > > > > > > On Apr 11, 2023, at 11:31, Cy Schubert <Cy.Schubert@cschubert.com> > wrote: > > > >=20 > > > > =EF=BB=BFIn message > <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn. > > d= > > > e>,=20 > > > > FreeBSD Us > > > > er writes: > > > >> Am Sun, 9 Apr 2023 14:37:03 +0200 > > > >> Mateusz Guzik <mjguzik@gmail.com> schrieb: > > > >>=20 > > > >>>> On 4/9/23, FreeBSD User <freebsd@walstatt-de.de> wrote: > > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 > main-n262052-0d4038 > > e= > > > 301 > > > >>> 2b: > > > >>>>> Sun Apr 9 > > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via > > > >>>>>=20 > > > >>>>> zpool upgrade POOLNAME > > > >>>>>=20 > > > >>>>> some boxes keep crashing when starting compiler runs (the > trigger is > > > >>>>> different on boxes). > > > >>>>>=20 > > > >>>>> ZFS module is statically compiled into the kernel (if this is of > > > >>>>> importance) > > > >>>>>=20 > > > >>>>> Last known good was: > > > >>>>>=20 > > > >>>>> [...] > > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 > > > >>>>> main-n262051-75379ea2e461: Sun Apr > > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: > > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 > 07:10:04 > > < > > > = > > > 0. > > > >>> 2> > > > >>>>> thor kernel: > > > >>>>> FreeBSD clang version 15.0.7 ( > https://github.com/llvm/llvm-project.gi > > t= > > > > > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor > kernel: > > > >>>>> VT(efifb): resolution > > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl > already > > > >>>>> present! > > > >>>>> [...] > > > >>>>>=20 > > > >>>>> The file /var/crash/info.X > > > >>>>>=20 > > > >>>>> contains: > > > >>>>>=20 > > > >>>>> [...] > > > >>>>>=20 > > > >>>>> root@thor:/var/crash # more info.2 > > > >>>>> Dump header from device: /dev/gpt/swap > > > >>>>> Architecture: amd64 > > > >>>>> Architecture Version: 2 > > > >>>>> Dump Length: 1095192576 > > > >>>>> Blocksize: 512 > > > >>>>> Compression: none > > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000 > > > >>>>> Hostname: thor.local > > > >>>>> Magic: FreeBSD Kernel Dump > > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 > main-n262052-0d4038e3012b: S > > u= > > > n=20 > > > >>> Apr > > > >>>>> 9 12:01:02 CEST > > > >>>>> 2023 > > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR > > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed > > > >>>>>=20 > > > >>>>> Dump Parity: 2961465682 > > > >>>>> Bounds: 2 > > > >>>>> Dump Status: good > > > >>>>>=20 > > > >>>>> Until reconfigured for more debug stuff I do not have more to > present > > .= > > > > > > >>>>>=20 > > > >>>>> I rememeber now really scraed that there was a HEADSUP in the > list re > > g= > > > ard > > > >>> ing > > > >>>>> some serious ZFS > > > >>>>> problems - I didn't find it right now. > > > >>>>>=20 > > > >>>>> Thanks in advance, > > > >>>>>=20 > > > >>>=20 > > > >>> That's fallout from the new block cloning feature, adding the > author > > > >>>=20 > > > >>=20 > > > >> Thanks. > > > >>=20 > > > >> As of this moment, all systems with the newest kernel and the new > ZFS op > > t= > > > ion=20 > > > >> enabled, crash - > > > >> the reason is mostly in different ZFS datasets. I guess there is > no way > > b > > > = > > > ack > > > >> once this faulty > > > >> option is enabled? > > > >=20 > > > > I've run a test on a scratch pool here, first without > block_cloning=20 > > > > enabled, then with. There was no corruption when block_cloning was=20 > > > > disabled. There was corruption when block_cloning was enabled. > > > >=20 > > > > I don't know of any way to revert back nor is there any way to fix > or=20 > > > > recover the corrupted blocks. > > > > > > Is the corruption still present after EXDEV fixes? > > > > Yes and no. > > > > Yes, there is corruption when block_cloning is enabled. > > > > There is no corruption when block_cloning is disabled. > > I should add some detail to this. > > The corruption experienced when block cloning is disabled was fixed by: > > - eb1feadc201a > - e2d997d1cbb9 > - d012836fb616 (specifically this commit) > - 20be1b4fc4b7 > > When block_cloning is enabled, the pool is corrupted. This has not been > fixed. > > > -- > Cheers, > Cy Schubert <Cy.Schubert@cschubert.com> > FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org > NTP: <cy@nwtime.org> Web: https://nwtime.org > > e^(i*pi)+1=0 > > > > -- Nuno Teixeira FreeBSD Committer (ports) [-- Attachment #2 --] <div dir="ltr"><div>Hello all,</div><div><br></div><div>at current 3fdb40d1befe after `zfs upgrade XXX`:<br></div><div><br></div><div>same problem when running compiler:</div><div><br></div><div>- poudriere: crash without dump</div><div>- make buildworld (/usr/src): shutdown -p (I will try to get a photo)</div><div><br></div><div>Is there a way to disable block clone?<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com">Cy.Schubert@cschubert.com</a>> escreveu no dia terça, 11/04/2023 à(s) 15:47:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">In message <<a href="mailto:20230411142831.DB8245FA@slippy.cwsent.com" target="_blank">20230411142831.DB8245FA@slippy.cwsent.com</a>>, Cy Schubert writes:<br> > In message <<a href="mailto:434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net" target="_blank">434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net</a>>, <br> > =?utf-8?Q?Pawe=C<br> > 5=82_Jakub_Dawidek?= writes:<br> > > <br> > ><br> > > > On Apr 11, 2023, at 11:31, Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com" target="_blank">Cy.Schubert@cschubert.com</a>> wrote:<br> > > >=20<br> > > > =EF=BB=BFIn message <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn.<br> > d=<br> > > e>,=20<br> > > > FreeBSD Us<br> > > > er writes:<br> > > >> Am Sun, 9 Apr 2023 14:37:03 +0200<br> > > >> Mateusz Guzik <<a href="mailto:mjguzik@gmail.com" target="_blank">mjguzik@gmail.com</a>> schrieb:<br> > > >>=20<br> > > >>>> On 4/9/23, FreeBSD User <<a href="mailto:freebsd@walstatt-de.de" target="_blank">freebsd@walstatt-de.de</a>> wrote:<br> > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 main-n262052-0d4038<br> > e=<br> > > 301<br> > > >>> 2b:<br> > > >>>>> Sun Apr 9<br> > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via<br> > > >>>>>=20<br> > > >>>>> zpool upgrade POOLNAME<br> > > >>>>>=20<br> > > >>>>> some boxes keep crashing when starting compiler runs (the trigger is<br> > > >>>>> different on boxes).<br> > > >>>>>=20<br> > > >>>>> ZFS module is statically compiled into the kernel (if this is of<br> > > >>>>> importance)<br> > > >>>>>=20<br> > > >>>>> Last known good was:<br> > > >>>>>=20<br> > > >>>>> [...]<br> > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7<br> > > >>>>> main-n262051-75379ea2e461: Sun Apr<br> > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel:<br> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 07:10:04<br> > <<br> > > =<br> > > 0.<br> > > >>> 2><br> > > >>>>> thor kernel:<br> > > >>>>> FreeBSD clang version 15.0.7 (<a href="https://github.com/llvm/llvm-project.gi" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project.gi</a><br>; > t=<br> > ><br> > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor kernel:<br> > > >>>>> VT(efifb): resolution<br> > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl already<br> > > >>>>> present!<br> > > >>>>> [...]<br> > > >>>>>=20<br> > > >>>>> The file /var/crash/info.X<br> > > >>>>>=20<br> > > >>>>> contains:<br> > > >>>>>=20<br> > > >>>>> [...]<br> > > >>>>>=20<br> > > >>>>> root@thor:/var/crash # more info.2<br> > > >>>>> Dump header from device: /dev/gpt/swap<br> > > >>>>> Architecture: amd64<br> > > >>>>> Architecture Version: 2<br> > > >>>>> Dump Length: 1095192576<br> > > >>>>> Blocksize: 512<br> > > >>>>> Compression: none<br> > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000<br> > > >>>>> Hostname: thor.local<br> > > >>>>> Magic: FreeBSD Kernel Dump<br> > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 main-n262052-0d4038e3012b: S<br> > u=<br> > > n=20<br> > > >>> Apr<br> > > >>>>> 9 12:01:02 CEST<br> > > >>>>> 2023<br> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR<br> > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed<br> > > >>>>>=20<br> > > >>>>> Dump Parity: 2961465682<br> > > >>>>> Bounds: 2<br> > > >>>>> Dump Status: good<br> > > >>>>>=20<br> > > >>>>> Until reconfigured for more debug stuff I do not have more to present<br> > .=<br> > ><br> > > >>>>>=20<br> > > >>>>> I rememeber now really scraed that there was a HEADSUP in the list re<br> > g=<br> > > ard<br> > > >>> ing<br> > > >>>>> some serious ZFS<br> > > >>>>> problems - I didn't find it right now.<br> > > >>>>>=20<br> > > >>>>> Thanks in advance,<br> > > >>>>>=20<br> > > >>>=20<br> > > >>> That's fallout from the new block cloning feature, adding the author<br> > > >>>=20<br> > > >>=20<br> > > >> Thanks.<br> > > >>=20<br> > > >> As of this moment, all systems with the newest kernel and the new ZFS op<br> > t=<br> > > ion=20<br> > > >> enabled, crash -<br> > > >> the reason is mostly in different ZFS datasets. I guess there is no way<br> > b<br> > > =<br> > > ack<br> > > >> once this faulty<br> > > >> option is enabled?<br> > > >=20<br> > > > I've run a test on a scratch pool here, first without block_cloning=20<br> > > > enabled, then with. There was no corruption when block_cloning was=20<br> > > > disabled. There was corruption when block_cloning was enabled.<br> > > >=20<br> > > > I don't know of any way to revert back nor is there any way to fix or=20<br> > > > recover the corrupted blocks.<br> > ><br> > > Is the corruption still present after EXDEV fixes?<br> ><br> > Yes and no.<br> ><br> > Yes, there is corruption when block_cloning is enabled.<br> ><br> > There is no corruption when block_cloning is disabled.<br> <br> I should add some detail to this.<br> <br> The corruption experienced when block cloning is disabled was fixed by:<br> <br> - eb1feadc201a<br> - e2d997d1cbb9<br> - d012836fb616 (specifically this commit)<br> - 20be1b4fc4b7<br> <br> When block_cloning is enabled, the pool is corrupted. This has not been <br> fixed.<br> <br> <br> -- <br> Cheers,<br> Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com" target="_blank">Cy.Schubert@cschubert.com</a>><br> FreeBSD UNIX: <cy@FreeBSD.org> Web: <a href="https://FreeBSD.org" rel="noreferrer" target="_blank">https://FreeBSD.org</a><br>; NTP: <<a href="mailto:cy@nwtime.org" target="_blank">cy@nwtime.org</a>> Web: <a href="https://nwtime.org" rel="noreferrer" target="_blank">https://nwtime.org</a><br>; <br> e^(i*pi)+1=0<br> <br> <br> <br> </blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span style="color:rgb(102,102,102)">Nuno Teixeira<br>FreeBSD Committer (ports)</span></div></div>help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFDf7UL6d-EyEhcPGjZfEia_qO9M%2BxY4bQMsD9hemmHYapQU4Q>