Date: Wed, 12 Apr 2023 14:16:07 +0100 From: Nuno Teixeira <eduardo@freebsd.org> To: Cy Schubert <Cy.Schubert@cschubert.com> Cc: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= <pawel@dawidek.net>, FreeBSD User <freebsd@walstatt-de.de>, Mateusz Guzik <mjguzik@gmail.com>, Pawel Jakub Dawidek <pjd@freebsd.org>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) Message-ID: <CAFDf7ULMjcD3NP8Q8uNv7uekcSyVFO2=WxFB2idMpiSveBWySA@mail.gmail.com> In-Reply-To: <CAFDf7UL6d-EyEhcPGjZfEia_qO9M%2BxY4bQMsD9hemmHYapQU4Q@mail.gmail.com> References: <20230411021919.0718F306@slippy.cwsent.com> <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> <20230411142831.DB8245FA@slippy.cwsent.com> <20230411144713.A94EA5FE@slippy.cwsent.com> <CAFDf7UL6d-EyEhcPGjZfEia_qO9M%2BxY4bQMsD9hemmHYapQU4Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000e80a0f05f9236911 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable (...) Trying : `zpool set feature@block_cloning=3Ddisabled zroot`: cannot set property for 'zroot': property 'feature@block_cloning' can only be set to 'disabled' at creation time Nuno Teixeira <eduardo@freebsd.org> escreveu no dia quarta, 12/04/2023 =C3= =A0(s) 13:57: > Hello all, > > at current 3fdb40d1befe after `zfs upgrade XXX`: > > same problem when running compiler: > > - poudriere: crash without dump > - make buildworld (/usr/src): shutdown -p (I will try to get a photo) > > Is there a way to disable block clone? > > Cy Schubert <Cy.Schubert@cschubert.com> escreveu no dia ter=C3=A7a, 11/04= /2023 > =C3=A0(s) 15:47: > >> In message <20230411142831.DB8245FA@slippy.cwsent.com>, Cy Schubert >> writes: >> > In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net>, >> > =3D?utf-8?Q?Pawe=3DC >> > 5=3D82_Jakub_Dawidek?=3D writes: >> > > >> > > >> > > > On Apr 11, 2023, at 11:31, Cy Schubert <Cy.Schubert@cschubert.com> >> wrote: >> > > >=3D20 >> > > > =3DEF=3DBB=3DBFIn message >> <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn. >> > d=3D >> > > e>,=3D20 >> > > > FreeBSD Us >> > > > er writes: >> > > >> Am Sun, 9 Apr 2023 14:37:03 +0200 >> > > >> Mateusz Guzik <mjguzik@gmail.com> schrieb: >> > > >>=3D20 >> > > >>>> On 4/9/23, FreeBSD User <freebsd@walstatt-de.de> wrote: >> > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 >> main-n262052-0d4038 >> > e=3D >> > > 301 >> > > >>> 2b: >> > > >>>>> Sun Apr 9 >> > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via >> > > >>>>>=3D20 >> > > >>>>> zpool upgrade POOLNAME >> > > >>>>>=3D20 >> > > >>>>> some boxes keep crashing when starting compiler runs (the >> trigger is >> > > >>>>> different on boxes). >> > > >>>>>=3D20 >> > > >>>>> ZFS module is statically compiled into the kernel (if this is = of >> > > >>>>> importance) >> > > >>>>>=3D20 >> > > >>>>> Last known good was: >> > > >>>>>=3D20 >> > > >>>>> [...] >> > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 >> > > >>>>> main-n262051-75379ea2e461: Sun Apr >> > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: >> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 >> 07:10:04 >> > < >> > > =3D >> > > 0. >> > > >>> 2> >> > > >>>>> thor kernel: >> > > >>>>> FreeBSD clang version 15.0.7 ( >> https://github.com/llvm/llvm-project.gi >> > t=3D >> > > >> > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor >> kernel: >> > > >>>>> VT(efifb): resolution >> > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl >> already >> > > >>>>> present! >> > > >>>>> [...] >> > > >>>>>=3D20 >> > > >>>>> The file /var/crash/info.X >> > > >>>>>=3D20 >> > > >>>>> contains: >> > > >>>>>=3D20 >> > > >>>>> [...] >> > > >>>>>=3D20 >> > > >>>>> root@thor:/var/crash # more info.2 >> > > >>>>> Dump header from device: /dev/gpt/swap >> > > >>>>> Architecture: amd64 >> > > >>>>> Architecture Version: 2 >> > > >>>>> Dump Length: 1095192576 >> > > >>>>> Blocksize: 512 >> > > >>>>> Compression: none >> > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000 >> > > >>>>> Hostname: thor.local >> > > >>>>> Magic: FreeBSD Kernel Dump >> > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 >> main-n262052-0d4038e3012b: S >> > u=3D >> > > n=3D20 >> > > >>> Apr >> > > >>>>> 9 12:01:02 CEST >> > > >>>>> 2023 >> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR >> > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed >> > > >>>>>=3D20 >> > > >>>>> Dump Parity: 2961465682 >> > > >>>>> Bounds: 2 >> > > >>>>> Dump Status: good >> > > >>>>>=3D20 >> > > >>>>> Until reconfigured for more debug stuff I do not have more to >> present >> > .=3D >> > > >> > > >>>>>=3D20 >> > > >>>>> I rememeber now really scraed that there was a HEADSUP in the >> list re >> > g=3D >> > > ard >> > > >>> ing >> > > >>>>> some serious ZFS >> > > >>>>> problems - I didn't find it right now. >> > > >>>>>=3D20 >> > > >>>>> Thanks in advance, >> > > >>>>>=3D20 >> > > >>>=3D20 >> > > >>> That's fallout from the new block cloning feature, adding the >> author >> > > >>>=3D20 >> > > >>=3D20 >> > > >> Thanks. >> > > >>=3D20 >> > > >> As of this moment, all systems with the newest kernel and the new >> ZFS op >> > t=3D >> > > ion=3D20 >> > > >> enabled, crash - >> > > >> the reason is mostly in different ZFS datasets. I guess there is >> no way >> > b >> > > =3D >> > > ack >> > > >> once this faulty >> > > >> option is enabled? >> > > >=3D20 >> > > > I've run a test on a scratch pool here, first without >> block_cloning=3D20 >> > > > enabled, then with. There was no corruption when block_cloning >> was=3D20 >> > > > disabled. There was corruption when block_cloning was enabled. >> > > >=3D20 >> > > > I don't know of any way to revert back nor is there any way to fix >> or=3D20 >> > > > recover the corrupted blocks. >> > > >> > > Is the corruption still present after EXDEV fixes? >> > >> > Yes and no. >> > >> > Yes, there is corruption when block_cloning is enabled. >> > >> > There is no corruption when block_cloning is disabled. >> >> I should add some detail to this. >> >> The corruption experienced when block cloning is disabled was fixed by: >> >> - eb1feadc201a >> - e2d997d1cbb9 >> - d012836fb616 (specifically this commit) >> - 20be1b4fc4b7 >> >> When block_cloning is enabled, the pool is corrupted. This has not been >> fixed. >> >> >> -- >> Cheers, >> Cy Schubert <Cy.Schubert@cschubert.com> >> FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org >> NTP: <cy@nwtime.org> Web: https://nwtime.org >> >> e^(i*pi)+1=3D0 >> >> >> >> > > -- > Nuno Teixeira > FreeBSD Committer (ports) > --=20 Nuno Teixeira FreeBSD Committer (ports) --000000000000e80a0f05f9236911 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>(...)</div><div><br></div><div>Trying :</div><div><br= ></div><div>`zpool set feature@block_cloning=3Ddisabled zroot`:<br>cannot s= et property for 'zroot': property 'feature@block_cloning' c= an only be set to 'disabled' at creation time</div></div><br><div c= lass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">Nuno Teixeira &l= t;<a href=3D"mailto:eduardo@freebsd.org">eduardo@freebsd.org</a>> escrev= eu no dia quarta, 12/04/2023 =C3=A0(s) 13:57:<br></div><blockquote class=3D= "gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(2= 04,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Hello all,</div><div><b= r></div><div>at current 3fdb40d1befe after `zfs upgrade XXX`:<br></div><div= ><br></div><div>same problem when running compiler:</div><div><br></div><di= v>- poudriere: crash without dump</div><div>- make buildworld (/usr/src): s= hutdown -p (I will try to get a photo)</div><div><br></div><div>Is there a = way to disable block clone?<br></div></div><br><div class=3D"gmail_quote"><= div dir=3D"ltr" class=3D"gmail_attr">Cy Schubert <<a href=3D"mailto:Cy.S= chubert@cschubert.com" target=3D"_blank">Cy.Schubert@cschubert.com</a>> = escreveu no dia ter=C3=A7a, 11/04/2023 =C3=A0(s) 15:47:<br></div><blockquot= e class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px s= olid rgb(204,204,204);padding-left:1ex">In message <<a href=3D"mailto:20= 230411142831.DB8245FA@slippy.cwsent.com" target=3D"_blank">20230411142831.D= B8245FA@slippy.cwsent.com</a>>, Cy Schubert writes:<br> > In message <<a href=3D"mailto:434B83DB-F6BB-436F-8AA5-385730D20BB1@= dawidek.net" target=3D"_blank">434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek= .net</a>>, <br> > =3D?utf-8?Q?Pawe=3DC<br> > 5=3D82_Jakub_Dawidek?=3D writes:<br> > > <br> > ><br> > > > On Apr 11, 2023, at 11:31, Cy Schubert <<a href=3D"mailto= :Cy.Schubert@cschubert.com" target=3D"_blank">Cy.Schubert@cschubert.com</a>= > wrote:<br> > > >=3D20<br> > > > =3DEF=3DBB=3DBFIn message <20230409161436.5412fa6e@thor.i= ntern.walstatt.dynvpn.<br> > d=3D<br> > > e>,=3D20<br> > > > FreeBSD Us<br> > > > er writes:<br> > > >> Am Sun, 9 Apr 2023 14:37:03 +0200<br> > > >> Mateusz Guzik <<a href=3D"mailto:mjguzik@gmail.com" t= arget=3D"_blank">mjguzik@gmail.com</a>> schrieb:<br> > > >>=3D20<br> > > >>>> On 4/9/23, FreeBSD User <<a href=3D"mailto:fr= eebsd@walstatt-de.de" target=3D"_blank">freebsd@walstatt-de.de</a>> wrot= e:<br> > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRE= NT #8 main-n262052-0d4038<br> > e=3D<br> > > 301<br> > > >>> 2b:<br> > > >>>>> Sun Apr=C2=A0 9<br> > > >>>>> 12:01:02 CEST 2023=C2=A0 amd64, AND upgradin= g ZPOOLs via<br> > > >>>>>=3D20<br> > > >>>>> zpool upgrade POOLNAME<br> > > >>>>>=3D20<br> > > >>>>> some boxes keep crashing when starting compi= ler runs (the trigger is<br> > > >>>>> different on boxes).<br> > > >>>>>=3D20<br> > > >>>>> ZFS module is statically compiled into the k= ernel (if this is of<br> > > >>>>> importance)<br> > > >>>>>=3D20<br> > > >>>>> Last known good was:<br> > > >>>>>=3D20<br> > > >>>>> [...]<br> > > >>>>> Apr=C2=A0 9 07:10:04 <0.2> thor kernel= : FreeBSD 14.0-CURRENT #7<br> > > >>>>> main-n262051-75379ea2e461: Sun Apr<br> > > >>>>> 9 00:12:57 CEST 2023 Apr=C2=A0 9 07:10:04 &l= t;0.2> thor kernel:<br> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/T= HOR amd64 Apr=C2=A0 9 07:10:04<br> >=C2=A0 <<br> > > =3D<br> > > 0.<br> > > >>> 2><br> > > >>>>> thor kernel:<br> > > >>>>> FreeBSD clang version 15.0.7 (<a href=3D"htt= ps://github.com/llvm/llvm-project.gi" rel=3D"noreferrer" target=3D"_blank">= https://github.com/llvm/llvm-project.gi</a><br>; > t=3D<br> > ><br> > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr=C2=A0 9 = 07:10:04 <0.2> thor kernel:<br> > > >>>>> VT(efifb): resolution<br> > > >>>>> 2560x1440 Apr=C2=A0 9 07:10:04 <0.2> t= hor kernel: module zfsctrl already<br> > > >>>>> present!<br> > > >>>>> [...]<br> > > >>>>>=3D20<br> > > >>>>> The file /var/crash/info.X<br> > > >>>>>=3D20<br> > > >>>>> contains:<br> > > >>>>>=3D20<br> > > >>>>> [...]<br> > > >>>>>=3D20<br> > > >>>>> root@thor:/var/crash # more info.2<br> > > >>>>> Dump header from device: /dev/gpt/swap<br> > > >>>>>=C2=A0 Architecture: amd64<br> > > >>>>>=C2=A0 Architecture Version: 2<br> > > >>>>>=C2=A0 Dump Length: 1095192576<br> > > >>>>>=C2=A0 Blocksize: 512<br> > > >>>>>=C2=A0 Compression: none<br> > > >>>>>=C2=A0 Dumptime: 2023-04-09 11:43:41 +0000<br= > > > >>>>>=C2=A0 Hostname: thor.local<br> > > >>>>>=C2=A0 Magic: FreeBSD Kernel Dump<br> > > >>>>>=C2=A0 Version String: FreeBSD 14.0-CURRENT #= 8 main-n262052-0d4038e3012b: S<br> > u=3D<br> > > n=3D20<br> > > >>> Apr<br> > > >>>>> 9 12:01:02 CEST<br> > > >>>>> 2023<br> > > >>>>>=C2=A0 =C2=A0 root@thor:/usr/obj/usr/src/amd6= 4.amd64/sys/THOR<br> > > >>>>>=C2=A0 Panic String: VERIFY(!zil_replaying(zi= log, tx)) failed<br> > > >>>>>=3D20<br> > > >>>>>=C2=A0 Dump Parity: 2961465682<br> > > >>>>>=C2=A0 Bounds: 2<br> > > >>>>>=C2=A0 Dump Status: good<br> > > >>>>>=3D20<br> > > >>>>> Until reconfigured for more debug stuff I do= not have more to present<br> > .=3D<br> > ><br> > > >>>>>=3D20<br> > > >>>>> I rememeber now really scraed that there was= a HEADSUP in the list re<br> > g=3D<br> > > ard<br> > > >>> ing<br> > > >>>>> some serious ZFS<br> > > >>>>> problems - I didn't find it right now.<b= r> > > >>>>>=3D20<br> > > >>>>> Thanks in advance,<br> > > >>>>>=3D20<br> > > >>>=3D20<br> > > >>> That's fallout from the new block cloning featur= e, adding the author<br> > > >>>=3D20<br> > > >>=3D20<br> > > >> Thanks.<br> > > >>=3D20<br> > > >> As of this moment, all systems with the newest kernel an= d the new ZFS op<br> > t=3D<br> > > ion=3D20<br> > > >> enabled, crash -<br> > > >> the reason is mostly in=C2=A0 different ZFS datasets. I = guess there is no way<br> >=C2=A0 b<br> > > =3D<br> > > ack<br> > > >> once this faulty<br> > > >> option is enabled?<br> > > >=3D20<br> > > > I've run a test on a scratch pool here, first without bl= ock_cloning=3D20<br> > > > enabled, then with. There was no corruption when block_cloni= ng was=3D20<br> > > > disabled. There was corruption when block_cloning was enable= d.<br> > > >=3D20<br> > > > I don't know of any way to revert back nor is there any = way to fix or=3D20<br> > > > recover the corrupted blocks.<br> > ><br> > > Is the corruption still present after EXDEV fixes?<br> ><br> > Yes and no.<br> ><br> > Yes, there is corruption when block_cloning is enabled.<br> ><br> > There is no corruption when block_cloning is disabled.<br> <br> I should add some detail to this.<br> <br> The corruption experienced when block cloning is disabled was fixed by:<br> <br> - eb1feadc201a<br> - e2d997d1cbb9<br> - d012836fb616 (specifically this commit)<br> - 20be1b4fc4b7<br> <br> When block_cloning is enabled, the pool is corrupted. This has not been <br= > fixed.<br> <br> <br> -- <br> Cheers,<br> Cy Schubert <<a href=3D"mailto:Cy.Schubert@cschubert.com" target=3D"_bla= nk">Cy.Schubert@cschubert.com</a>><br> FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 <a href= =3D"https://FreeBSD.org" rel=3D"noreferrer" target=3D"_blank">https://FreeB= SD.org</a><br> NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:cy@nwtim= e.org" target=3D"_blank">cy@nwtime.org</a>>=C2=A0 =C2=A0 Web:=C2=A0 <a h= ref=3D"https://nwtime.org" rel=3D"noreferrer" target=3D"_blank">https://nwt= ime.org</a><br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 e^(i*pi)+1=3D0<br> <br> <br> <br> </blockquote></div><br clear=3D"all"><br><span>-- </span><br><div dir=3D"lt= r"><div dir=3D"ltr"><span style=3D"color:rgb(102,102,102)">Nuno Teixeira<br= >FreeBSD Committer (ports)</span></div></div> </blockquote></div><br clear=3D"all"><br><span class=3D"gmail_signature_pre= fix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"l= tr"><span style=3D"color:rgb(102,102,102)">Nuno Teixeira<br>FreeBSD Committ= er (ports)</span></div></div> --000000000000e80a0f05f9236911--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFDf7ULMjcD3NP8Q8uNv7uekcSyVFO2=WxFB2idMpiSveBWySA>