Date: Wed, 12 Apr 2023 13:57:02 +0100 From: Nuno Teixeira <eduardo@freebsd.org> To: Cy Schubert <Cy.Schubert@cschubert.com> Cc: =?UTF-8?Q?Pawe=C5=82_Jakub_Dawidek?= <pawel@dawidek.net>, FreeBSD User <freebsd@walstatt-de.de>, Mateusz Guzik <mjguzik@gmail.com>, Pawel Jakub Dawidek <pjd@freebsd.org>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: CURRENT: Panic VERIFY(!zil_replaying(zilog, tx)) failed (and crashing) Message-ID: <CAFDf7UL6d-EyEhcPGjZfEia_qO9M%2BxY4bQMsD9hemmHYapQU4Q@mail.gmail.com> In-Reply-To: <20230411144713.A94EA5FE@slippy.cwsent.com> References: <20230411021919.0718F306@slippy.cwsent.com> <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net> <20230411142831.DB8245FA@slippy.cwsent.com> <20230411144713.A94EA5FE@slippy.cwsent.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000a5d6f005f92325c6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello all, at current 3fdb40d1befe after `zfs upgrade XXX`: same problem when running compiler: - poudriere: crash without dump - make buildworld (/usr/src): shutdown -p (I will try to get a photo) Is there a way to disable block clone? Cy Schubert <Cy.Schubert@cschubert.com> escreveu no dia ter=C3=A7a, 11/04/2= 023 =C3=A0(s) 15:47: > In message <20230411142831.DB8245FA@slippy.cwsent.com>, Cy Schubert > writes: > > In message <434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek.net>, > > =3D?utf-8?Q?Pawe=3DC > > 5=3D82_Jakub_Dawidek?=3D writes: > > > > > > > > > > On Apr 11, 2023, at 11:31, Cy Schubert <Cy.Schubert@cschubert.com> > wrote: > > > >=3D20 > > > > =3DEF=3DBB=3DBFIn message > <20230409161436.5412fa6e@thor.intern.walstatt.dynvpn. > > d=3D > > > e>,=3D20 > > > > FreeBSD Us > > > > er writes: > > > >> Am Sun, 9 Apr 2023 14:37:03 +0200 > > > >> Mateusz Guzik <mjguzik@gmail.com> schrieb: > > > >>=3D20 > > > >>>> On 4/9/23, FreeBSD User <freebsd@walstatt-de.de> wrote: > > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRENT #8 > main-n262052-0d4038 > > e=3D > > > 301 > > > >>> 2b: > > > >>>>> Sun Apr 9 > > > >>>>> 12:01:02 CEST 2023 amd64, AND upgrading ZPOOLs via > > > >>>>>=3D20 > > > >>>>> zpool upgrade POOLNAME > > > >>>>>=3D20 > > > >>>>> some boxes keep crashing when starting compiler runs (the > trigger is > > > >>>>> different on boxes). > > > >>>>>=3D20 > > > >>>>> ZFS module is statically compiled into the kernel (if this is o= f > > > >>>>> importance) > > > >>>>>=3D20 > > > >>>>> Last known good was: > > > >>>>>=3D20 > > > >>>>> [...] > > > >>>>> Apr 9 07:10:04 <0.2> thor kernel: FreeBSD 14.0-CURRENT #7 > > > >>>>> main-n262051-75379ea2e461: Sun Apr > > > >>>>> 9 00:12:57 CEST 2023 Apr 9 07:10:04 <0.2> thor kernel: > > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR amd64 Apr 9 > 07:10:04 > > < > > > =3D > > > 0. > > > >>> 2> > > > >>>>> thor kernel: > > > >>>>> FreeBSD clang version 15.0.7 ( > https://github.com/llvm/llvm-project.gi > > t=3D > > > > > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr 9 07:10:04 <0.2> thor > kernel: > > > >>>>> VT(efifb): resolution > > > >>>>> 2560x1440 Apr 9 07:10:04 <0.2> thor kernel: module zfsctrl > already > > > >>>>> present! > > > >>>>> [...] > > > >>>>>=3D20 > > > >>>>> The file /var/crash/info.X > > > >>>>>=3D20 > > > >>>>> contains: > > > >>>>>=3D20 > > > >>>>> [...] > > > >>>>>=3D20 > > > >>>>> root@thor:/var/crash # more info.2 > > > >>>>> Dump header from device: /dev/gpt/swap > > > >>>>> Architecture: amd64 > > > >>>>> Architecture Version: 2 > > > >>>>> Dump Length: 1095192576 > > > >>>>> Blocksize: 512 > > > >>>>> Compression: none > > > >>>>> Dumptime: 2023-04-09 11:43:41 +0000 > > > >>>>> Hostname: thor.local > > > >>>>> Magic: FreeBSD Kernel Dump > > > >>>>> Version String: FreeBSD 14.0-CURRENT #8 > main-n262052-0d4038e3012b: S > > u=3D > > > n=3D20 > > > >>> Apr > > > >>>>> 9 12:01:02 CEST > > > >>>>> 2023 > > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/THOR > > > >>>>> Panic String: VERIFY(!zil_replaying(zilog, tx)) failed > > > >>>>>=3D20 > > > >>>>> Dump Parity: 2961465682 > > > >>>>> Bounds: 2 > > > >>>>> Dump Status: good > > > >>>>>=3D20 > > > >>>>> Until reconfigured for more debug stuff I do not have more to > present > > .=3D > > > > > > >>>>>=3D20 > > > >>>>> I rememeber now really scraed that there was a HEADSUP in the > list re > > g=3D > > > ard > > > >>> ing > > > >>>>> some serious ZFS > > > >>>>> problems - I didn't find it right now. > > > >>>>>=3D20 > > > >>>>> Thanks in advance, > > > >>>>>=3D20 > > > >>>=3D20 > > > >>> That's fallout from the new block cloning feature, adding the > author > > > >>>=3D20 > > > >>=3D20 > > > >> Thanks. > > > >>=3D20 > > > >> As of this moment, all systems with the newest kernel and the new > ZFS op > > t=3D > > > ion=3D20 > > > >> enabled, crash - > > > >> the reason is mostly in different ZFS datasets. I guess there is > no way > > b > > > =3D > > > ack > > > >> once this faulty > > > >> option is enabled? > > > >=3D20 > > > > I've run a test on a scratch pool here, first without > block_cloning=3D20 > > > > enabled, then with. There was no corruption when block_cloning was= =3D20 > > > > disabled. There was corruption when block_cloning was enabled. > > > >=3D20 > > > > I don't know of any way to revert back nor is there any way to fix > or=3D20 > > > > recover the corrupted blocks. > > > > > > Is the corruption still present after EXDEV fixes? > > > > Yes and no. > > > > Yes, there is corruption when block_cloning is enabled. > > > > There is no corruption when block_cloning is disabled. > > I should add some detail to this. > > The corruption experienced when block cloning is disabled was fixed by: > > - eb1feadc201a > - e2d997d1cbb9 > - d012836fb616 (specifically this commit) > - 20be1b4fc4b7 > > When block_cloning is enabled, the pool is corrupted. This has not been > fixed. > > > -- > Cheers, > Cy Schubert <Cy.Schubert@cschubert.com> > FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org > NTP: <cy@nwtime.org> Web: https://nwtime.org > > e^(i*pi)+1=3D0 > > > > --=20 Nuno Teixeira FreeBSD Committer (ports) --000000000000a5d6f005f92325c6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hello all,</div><div><br></div><div>at current 3fdb40= d1befe after `zfs upgrade XXX`:<br></div><div><br></div><div>same problem w= hen running compiler:</div><div><br></div><div>- poudriere: crash without d= ump</div><div>- make buildworld (/usr/src): shutdown -p (I will try to get = a photo)</div><div><br></div><div>Is there a way to disable block clone?<br= ></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail= _attr">Cy Schubert <<a href=3D"mailto:Cy.Schubert@cschubert.com">Cy.Schu= bert@cschubert.com</a>> escreveu no dia ter=C3=A7a, 11/04/2023 =C3=A0(s)= 15:47:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">In messa= ge <<a href=3D"mailto:20230411142831.DB8245FA@slippy.cwsent.com" target= =3D"_blank">20230411142831.DB8245FA@slippy.cwsent.com</a>>, Cy Schubert = writes:<br> > In message <<a href=3D"mailto:434B83DB-F6BB-436F-8AA5-385730D20BB1@= dawidek.net" target=3D"_blank">434B83DB-F6BB-436F-8AA5-385730D20BB1@dawidek= .net</a>>, <br> > =3D?utf-8?Q?Pawe=3DC<br> > 5=3D82_Jakub_Dawidek?=3D writes:<br> > > <br> > ><br> > > > On Apr 11, 2023, at 11:31, Cy Schubert <<a href=3D"mailto= :Cy.Schubert@cschubert.com" target=3D"_blank">Cy.Schubert@cschubert.com</a>= > wrote:<br> > > >=3D20<br> > > > =3DEF=3DBB=3DBFIn message <20230409161436.5412fa6e@thor.i= ntern.walstatt.dynvpn.<br> > d=3D<br> > > e>,=3D20<br> > > > FreeBSD Us<br> > > > er writes:<br> > > >> Am Sun, 9 Apr 2023 14:37:03 +0200<br> > > >> Mateusz Guzik <<a href=3D"mailto:mjguzik@gmail.com" t= arget=3D"_blank">mjguzik@gmail.com</a>> schrieb:<br> > > >>=3D20<br> > > >>>> On 4/9/23, FreeBSD User <<a href=3D"mailto:fr= eebsd@walstatt-de.de" target=3D"_blank">freebsd@walstatt-de.de</a>> wrot= e:<br> > > >>>>> Today, after upgrading to FreeBSD 14.0-CURRE= NT #8 main-n262052-0d4038<br> > e=3D<br> > > 301<br> > > >>> 2b:<br> > > >>>>> Sun Apr=C2=A0 9<br> > > >>>>> 12:01:02 CEST 2023=C2=A0 amd64, AND upgradin= g ZPOOLs via<br> > > >>>>>=3D20<br> > > >>>>> zpool upgrade POOLNAME<br> > > >>>>>=3D20<br> > > >>>>> some boxes keep crashing when starting compi= ler runs (the trigger is<br> > > >>>>> different on boxes).<br> > > >>>>>=3D20<br> > > >>>>> ZFS module is statically compiled into the k= ernel (if this is of<br> > > >>>>> importance)<br> > > >>>>>=3D20<br> > > >>>>> Last known good was:<br> > > >>>>>=3D20<br> > > >>>>> [...]<br> > > >>>>> Apr=C2=A0 9 07:10:04 <0.2> thor kernel= : FreeBSD 14.0-CURRENT #7<br> > > >>>>> main-n262051-75379ea2e461: Sun Apr<br> > > >>>>> 9 00:12:57 CEST 2023 Apr=C2=A0 9 07:10:04 &l= t;0.2> thor kernel:<br> > > >>>>> root@thor:/usr/obj/usr/src/amd64.amd64/sys/T= HOR amd64 Apr=C2=A0 9 07:10:04<br> >=C2=A0 <<br> > > =3D<br> > > 0.<br> > > >>> 2><br> > > >>>>> thor kernel:<br> > > >>>>> FreeBSD clang version 15.0.7 (<a href=3D"htt= ps://github.com/llvm/llvm-project.gi" rel=3D"noreferrer" target=3D"_blank">= https://github.com/llvm/llvm-project.gi</a><br>; > t=3D<br> > ><br> > > >>>>> llvmorg-15.0.7-0-g8dfdcc7b7bf6) Apr=C2=A0 9 = 07:10:04 <0.2> thor kernel:<br> > > >>>>> VT(efifb): resolution<br> > > >>>>> 2560x1440 Apr=C2=A0 9 07:10:04 <0.2> t= hor kernel: module zfsctrl already<br> > > >>>>> present!<br> > > >>>>> [...]<br> > > >>>>>=3D20<br> > > >>>>> The file /var/crash/info.X<br> > > >>>>>=3D20<br> > > >>>>> contains:<br> > > >>>>>=3D20<br> > > >>>>> [...]<br> > > >>>>>=3D20<br> > > >>>>> root@thor:/var/crash # more info.2<br> > > >>>>> Dump header from device: /dev/gpt/swap<br> > > >>>>>=C2=A0 Architecture: amd64<br> > > >>>>>=C2=A0 Architecture Version: 2<br> > > >>>>>=C2=A0 Dump Length: 1095192576<br> > > >>>>>=C2=A0 Blocksize: 512<br> > > >>>>>=C2=A0 Compression: none<br> > > >>>>>=C2=A0 Dumptime: 2023-04-09 11:43:41 +0000<br= > > > >>>>>=C2=A0 Hostname: thor.local<br> > > >>>>>=C2=A0 Magic: FreeBSD Kernel Dump<br> > > >>>>>=C2=A0 Version String: FreeBSD 14.0-CURRENT #= 8 main-n262052-0d4038e3012b: S<br> > u=3D<br> > > n=3D20<br> > > >>> Apr<br> > > >>>>> 9 12:01:02 CEST<br> > > >>>>> 2023<br> > > >>>>>=C2=A0 =C2=A0 root@thor:/usr/obj/usr/src/amd6= 4.amd64/sys/THOR<br> > > >>>>>=C2=A0 Panic String: VERIFY(!zil_replaying(zi= log, tx)) failed<br> > > >>>>>=3D20<br> > > >>>>>=C2=A0 Dump Parity: 2961465682<br> > > >>>>>=C2=A0 Bounds: 2<br> > > >>>>>=C2=A0 Dump Status: good<br> > > >>>>>=3D20<br> > > >>>>> Until reconfigured for more debug stuff I do= not have more to present<br> > .=3D<br> > ><br> > > >>>>>=3D20<br> > > >>>>> I rememeber now really scraed that there was= a HEADSUP in the list re<br> > g=3D<br> > > ard<br> > > >>> ing<br> > > >>>>> some serious ZFS<br> > > >>>>> problems - I didn't find it right now.<b= r> > > >>>>>=3D20<br> > > >>>>> Thanks in advance,<br> > > >>>>>=3D20<br> > > >>>=3D20<br> > > >>> That's fallout from the new block cloning featur= e, adding the author<br> > > >>>=3D20<br> > > >>=3D20<br> > > >> Thanks.<br> > > >>=3D20<br> > > >> As of this moment, all systems with the newest kernel an= d the new ZFS op<br> > t=3D<br> > > ion=3D20<br> > > >> enabled, crash -<br> > > >> the reason is mostly in=C2=A0 different ZFS datasets. I = guess there is no way<br> >=C2=A0 b<br> > > =3D<br> > > ack<br> > > >> once this faulty<br> > > >> option is enabled?<br> > > >=3D20<br> > > > I've run a test on a scratch pool here, first without bl= ock_cloning=3D20<br> > > > enabled, then with. There was no corruption when block_cloni= ng was=3D20<br> > > > disabled. There was corruption when block_cloning was enable= d.<br> > > >=3D20<br> > > > I don't know of any way to revert back nor is there any = way to fix or=3D20<br> > > > recover the corrupted blocks.<br> > ><br> > > Is the corruption still present after EXDEV fixes?<br> ><br> > Yes and no.<br> ><br> > Yes, there is corruption when block_cloning is enabled.<br> ><br> > There is no corruption when block_cloning is disabled.<br> <br> I should add some detail to this.<br> <br> The corruption experienced when block cloning is disabled was fixed by:<br> <br> - eb1feadc201a<br> - e2d997d1cbb9<br> - d012836fb616 (specifically this commit)<br> - 20be1b4fc4b7<br> <br> When block_cloning is enabled, the pool is corrupted. This has not been <br= > fixed.<br> <br> <br> -- <br> Cheers,<br> Cy Schubert <<a href=3D"mailto:Cy.Schubert@cschubert.com" target=3D"_bla= nk">Cy.Schubert@cschubert.com</a>><br> FreeBSD UNIX:=C2=A0 <cy@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 <a href= =3D"https://FreeBSD.org" rel=3D"noreferrer" target=3D"_blank">https://FreeB= SD.org</a><br> NTP:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:cy@nwtim= e.org" target=3D"_blank">cy@nwtime.org</a>>=C2=A0 =C2=A0 Web:=C2=A0 <a h= ref=3D"https://nwtime.org" rel=3D"noreferrer" target=3D"_blank">https://nwt= ime.org</a><br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 e^(i*pi)+1=3D0<br> <br> <br> <br> </blockquote></div><br clear=3D"all"><br><span class=3D"gmail_signature_pre= fix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"l= tr"><span style=3D"color:rgb(102,102,102)">Nuno Teixeira<br>FreeBSD Committ= er (ports)</span></div></div> --000000000000a5d6f005f92325c6--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFDf7UL6d-EyEhcPGjZfEia_qO9M%2BxY4bQMsD9hemmHYapQU4Q>