Date: Wed, 12 Apr 2023 14:16:07 +0100 From: Nuno Teixeira <eduardo@freebsd.org> To: Cy Schubert <Cy.Schubert@cschubert.com> Cc: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= <pawel@dawidek.net>, FreeBSD User <freebsd@walstatt-de.de>, Mateusz Guzik <mjguzik@gmail.com>, Pawel Jakub Dawidek <pjd@freebsd.org>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) Message-ID: <CAFDf7ULMjcD3NP8Q8uNv7uekcSyVFO2=WxFB2idMpiSveBWySA@mail.gmail.com> In-Reply-To: <CAFDf7UL6d-EyEhcPGjZfEia_qO9M%2BxY4bQMsD9hemmHYapQU4Q@mail.gmail.com> References: <20230411021919.0718F306@slippy.cwsent.com> <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> <20230411142831.DB8245FA@slippy.cwsent.com> <20230411144713.A94EA5FE@slippy.cwsent.com> <CAFDf7UL6d-EyEhcPGjZfEia_qO9M%2BxY4bQMsD9hemmHYapQU4Q@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] (...) Trying : `zpool set feature@block_cloning=disabled zroot`: cannot set property for 'zroot': property 'feature@block_cloning' can only be set to 'disabled' at creation time Nuno Teixeira <eduardo@freebsd.org> escreveu no dia quarta, 12/04/2023 à(s) 13:57: > Hello all, > > at current 3fdb40d1befe after `zfs upgrade XXX`: > > same problem when running compiler: > > - poudriere: crash without dump > - make buildworld (/usr/src): shutdown -p (I will try to get a photo) > > Is there a way to disable block clone? > > Cy Schubert <Cy.Schubert@cschubert.com> escreveu no dia terça, 11/04/2023 > à(s) 15:47: > >> In message <20230411142831.DB8245FA@slippy.cwsent.com>, Cy Schubert >> writes: >> > In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net>, >> > =?utf-8?Q?Pawe=C >> > 5=82_Jakub_Dawidek?= writes: >> > > >> > > >> > > > On Apr 11, 2023, at 11:31, Cy Schubert <Cy.Schubert@cschubert.com> >> wrote: >> > > >=20 >> > > > =EF=BB=BFIn message >> <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn. >> > d= >> > > e>,=20 >> > > > FreeBSD Us >> > > > er writes: >> > > >> Am Sun, 9 Apr 2023 14:37:03 +0200 >> > > >> Mateusz Guzik <mjguzik@gmail.com> schrieb: >> > > >>=20 >> > > >>>> On 4/9/23, FreeBSD User <freebsd@walstatt-de.de> wrote: >> > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 >> main-n262052-0d4038 >> > e= >> > > 301 >> > > >>> 2b: >> > > >>>>> Sun Apr 9 >> > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via >> > > >>>>>=20 >> > > >>>>> zpool upgrade POOLNAME >> > > >>>>>=20 >> > > >>>>> some boxes keep crashing when starting compiler runs (the >> trigger is >> > > >>>>> different on boxes). >> > > >>>>>=20 >> > > >>>>> ZFS module is statically compiled into the kernel (if this is of >> > > >>>>> importance) >> > > >>>>>=20 >> > > >>>>> Last known good was: >> > > >>>>>=20 >> > > >>>>> [...] >> > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 >> > > >>>>> main-n262051-75379ea2e461: Sun Apr >> > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: >> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 >> 07:10:04 >> > < >> > > = >> > > 0. >> > > >>> 2> >> > > >>>>> thor kernel: >> > > >>>>> FreeBSD clang version 15.0.7 ( >> https://github.com/llvm/llvm-project.gi >> > t= >> > > >> > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor >> kernel: >> > > >>>>> VT(efifb): resolution >> > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl >> already >> > > >>>>> present! >> > > >>>>> [...] >> > > >>>>>=20 >> > > >>>>> The file /var/crash/info.X >> > > >>>>>=20 >> > > >>>>> contains: >> > > >>>>>=20 >> > > >>>>> [...] >> > > >>>>>=20 >> > > >>>>> root@thor:/var/crash # more info.2 >> > > >>>>> Dump header from device: /dev/gpt/swap >> > > >>>>> Architecture: amd64 >> > > >>>>> Architecture Version: 2 >> > > >>>>> Dump Length: 1095192576 >> > > >>>>> Blocksize: 512 >> > > >>>>> Compression: none >> > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000 >> > > >>>>> Hostname: thor.local >> > > >>>>> Magic: FreeBSD Kernel Dump >> > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 >> main-n262052-0d4038e3012b: S >> > u= >> > > n=20 >> > > >>> Apr >> > > >>>>> 9 12:01:02 CEST >> > > >>>>> 2023 >> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR >> > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed >> > > >>>>>=20 >> > > >>>>> Dump Parity: 2961465682 >> > > >>>>> Bounds: 2 >> > > >>>>> Dump Status: good >> > > >>>>>=20 >> > > >>>>> Until reconfigured for more debug stuff I do not have more to >> present >> > .= >> > > >> > > >>>>>=20 >> > > >>>>> I rememeber now really scraed that there was a HEADSUP in the >> list re >> > g= >> > > ard >> > > >>> ing >> > > >>>>> some serious ZFS >> > > >>>>> problems - I didn't find it right now. >> > > >>>>>=20 >> > > >>>>> Thanks in advance, >> > > >>>>>=20 >> > > >>>=20 >> > > >>> That's fallout from the new block cloning feature, adding the >> author >> > > >>>=20 >> > > >>=20 >> > > >> Thanks. >> > > >>=20 >> > > >> As of this moment, all systems with the newest kernel and the new >> ZFS op >> > t= >> > > ion=20 >> > > >> enabled, crash - >> > > >> the reason is mostly in different ZFS datasets. I guess there is >> no way >> > b >> > > = >> > > ack >> > > >> once this faulty >> > > >> option is enabled? >> > > >=20 >> > > > I've run a test on a scratch pool here, first without >> block_cloning=20 >> > > > enabled, then with. There was no corruption when block_cloning >> was=20 >> > > > disabled. There was corruption when block_cloning was enabled. >> > > >=20 >> > > > I don't know of any way to revert back nor is there any way to fix >> or=20 >> > > > recover the corrupted blocks. >> > > >> > > Is the corruption still present after EXDEV fixes? >> > >> > Yes and no. >> > >> > Yes, there is corruption when block_cloning is enabled. >> > >> > There is no corruption when block_cloning is disabled. >> >> I should add some detail to this. >> >> The corruption experienced when block cloning is disabled was fixed by: >> >> - eb1feadc201a >> - e2d997d1cbb9 >> - d012836fb616 (specifically this commit) >> - 20be1b4fc4b7 >> >> When block_cloning is enabled, the pool is corrupted. This has not been >> fixed. >> >> >> -- >> Cheers, >> Cy Schubert <Cy.Schubert@cschubert.com> >> FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org >> NTP: <cy@nwtime.org> Web: https://nwtime.org >> >> e^(i*pi)+1=0 >> >> >> >> > > -- > Nuno Teixeira > FreeBSD Committer (ports) > -- Nuno Teixeira FreeBSD Committer (ports) [-- Attachment #2 --] <div dir="ltr"><div>(...)</div><div><br></div><div>Trying :</div><div><br></div><div>`zpool set feature@block_cloning=disabled zroot`:<br>cannot set property for 'zroot': property 'feature@block_cloning' can only be set to 'disabled' at creation time</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Nuno Teixeira <<a href="mailto:eduardo@freebsd.org">eduardo@freebsd.org</a>> escreveu no dia quarta, 12/04/2023 à(s) 13:57:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello all,</div><div><br></div><div>at current 3fdb40d1befe after `zfs upgrade XXX`:<br></div><div><br></div><div>same problem when running compiler:</div><div><br></div><div>- poudriere: crash without dump</div><div>- make buildworld (/usr/src): shutdown -p (I will try to get a photo)</div><div><br></div><div>Is there a way to disable block clone?<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com" target="_blank">Cy.Schubert@cschubert.com</a>> escreveu no dia terça, 11/04/2023 à(s) 15:47:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">In message <<a href="mailto:20230411142831.DB8245FA@slippy.cwsent.com" target="_blank">20230411142831.DB8245FA@slippy.cwsent.com</a>>, Cy Schubert writes:<br> > In message <<a href="mailto:434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net" target="_blank">434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net</a>>, <br> > =?utf-8?Q?Pawe=C<br> > 5=82_Jakub_Dawidek?= writes:<br> > > <br> > ><br> > > > On Apr 11, 2023, at 11:31, Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com" target="_blank">Cy.Schubert@cschubert.com</a>> wrote:<br> > > >=20<br> > > > =EF=BB=BFIn message <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn.<br> > d=<br> > > e>,=20<br> > > > FreeBSD Us<br> > > > er writes:<br> > > >> Am Sun, 9 Apr 2023 14:37:03 +0200<br> > > >> Mateusz Guzik <<a href="mailto:mjguzik@gmail.com" target="_blank">mjguzik@gmail.com</a>> schrieb:<br> > > >>=20<br> > > >>>> On 4/9/23, FreeBSD User <<a href="mailto:freebsd@walstatt-de.de" target="_blank">freebsd@walstatt-de.de</a>> wrote:<br> > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 main-n262052-0d4038<br> > e=<br> > > 301<br> > > >>> 2b:<br> > > >>>>> Sun Apr 9<br> > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via<br> > > >>>>>=20<br> > > >>>>> zpool upgrade POOLNAME<br> > > >>>>>=20<br> > > >>>>> some boxes keep crashing when starting compiler runs (the trigger is<br> > > >>>>> different on boxes).<br> > > >>>>>=20<br> > > >>>>> ZFS module is statically compiled into the kernel (if this is of<br> > > >>>>> importance)<br> > > >>>>>=20<br> > > >>>>> Last known good was:<br> > > >>>>>=20<br> > > >>>>> [...]<br> > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7<br> > > >>>>> main-n262051-75379ea2e461: Sun Apr<br> > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel:<br> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 07:10:04<br> > <<br> > > =<br> > > 0.<br> > > >>> 2><br> > > >>>>> thor kernel:<br> > > >>>>> FreeBSD clang version 15.0.7 (<a href="https://github.com/llvm/llvm-project.gi" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project.gi</a><br>; > t=<br> > ><br> > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor kernel:<br> > > >>>>> VT(efifb): resolution<br> > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl already<br> > > >>>>> present!<br> > > >>>>> [...]<br> > > >>>>>=20<br> > > >>>>> The file /var/crash/info.X<br> > > >>>>>=20<br> > > >>>>> contains:<br> > > >>>>>=20<br> > > >>>>> [...]<br> > > >>>>>=20<br> > > >>>>> root@thor:/var/crash # more info.2<br> > > >>>>> Dump header from device: /dev/gpt/swap<br> > > >>>>> Architecture: amd64<br> > > >>>>> Architecture Version: 2<br> > > >>>>> Dump Length: 1095192576<br> > > >>>>> Blocksize: 512<br> > > >>>>> Compression: none<br> > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000<br> > > >>>>> Hostname: thor.local<br> > > >>>>> Magic: FreeBSD Kernel Dump<br> > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 main-n262052-0d4038e3012b: S<br> > u=<br> > > n=20<br> > > >>> Apr<br> > > >>>>> 9 12:01:02 CEST<br> > > >>>>> 2023<br> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR<br> > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed<br> > > >>>>>=20<br> > > >>>>> Dump Parity: 2961465682<br> > > >>>>> Bounds: 2<br> > > >>>>> Dump Status: good<br> > > >>>>>=20<br> > > >>>>> Until reconfigured for more debug stuff I do not have more to present<br> > .=<br> > ><br> > > >>>>>=20<br> > > >>>>> I rememeber now really scraed that there was a HEADSUP in the list re<br> > g=<br> > > ard<br> > > >>> ing<br> > > >>>>> some serious ZFS<br> > > >>>>> problems - I didn't find it right now.<br> > > >>>>>=20<br> > > >>>>> Thanks in advance,<br> > > >>>>>=20<br> > > >>>=20<br> > > >>> That's fallout from the new block cloning feature, adding the author<br> > > >>>=20<br> > > >>=20<br> > > >> Thanks.<br> > > >>=20<br> > > >> As of this moment, all systems with the newest kernel and the new ZFS op<br> > t=<br> > > ion=20<br> > > >> enabled, crash -<br> > > >> the reason is mostly in different ZFS datasets. I guess there is no way<br> > b<br> > > =<br> > > ack<br> > > >> once this faulty<br> > > >> option is enabled?<br> > > >=20<br> > > > I've run a test on a scratch pool here, first without block_cloning=20<br> > > > enabled, then with. There was no corruption when block_cloning was=20<br> > > > disabled. There was corruption when block_cloning was enabled.<br> > > >=20<br> > > > I don't know of any way to revert back nor is there any way to fix or=20<br> > > > recover the corrupted blocks.<br> > ><br> > > Is the corruption still present after EXDEV fixes?<br> ><br> > Yes and no.<br> ><br> > Yes, there is corruption when block_cloning is enabled.<br> ><br> > There is no corruption when block_cloning is disabled.<br> <br> I should add some detail to this.<br> <br> The corruption experienced when block cloning is disabled was fixed by:<br> <br> - eb1feadc201a<br> - e2d997d1cbb9<br> - d012836fb616 (specifically this commit)<br> - 20be1b4fc4b7<br> <br> When block_cloning is enabled, the pool is corrupted. This has not been <br> fixed.<br> <br> <br> -- <br> Cheers,<br> Cy Schubert <<a href="mailto:Cy.Schubert@cschubert.com" target="_blank">Cy.Schubert@cschubert.com</a>><br> FreeBSD UNIX: <cy@FreeBSD.org> Web: <a href="https://FreeBSD.org" rel="noreferrer" target="_blank">https://FreeBSD.org</a><br>; NTP: <<a href="mailto:cy@nwtime.org" target="_blank">cy@nwtime.org</a>> Web: <a href="https://nwtime.org" rel="noreferrer" target="_blank">https://nwtime.org</a><br>; <br> e^(i*pi)+1=0<br> <br> <br> <br> </blockquote></div><br clear="all"><br><span>-- </span><br><div dir="ltr"><div dir="ltr"><span style="color:rgb(102,102,102)">Nuno Teixeira<br>FreeBSD Committer (ports)</span></div></div> </blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span style="color:rgb(102,102,102)">Nuno Teixeira<br>FreeBSD Committer (ports)</span></div></div>help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFDf7ULMjcD3NP8Q8uNv7uekcSyVFO2=WxFB2idMpiSveBWySA>