Date: Sat, 15 Apr 2023 09:09:28 -0700 From: Cy Schubert <Cy.Schubert@cschubert.com> To: FreeBSD User <freebsd@walstatt-de.de> Cc: Cy Schubert <Cy.Schubert@cschubert.com>, Mark Millard <marklmi@yahoo.com>, Charlie Li <vishwin@freebsd.org>, Pawel Jakub Dawidek <pjd@FreeBSD.org>, Mateusz Guzik <mjguzik@gmail.com>, dev-commits-src-main@freebsd.org, Current FreeBSD <freebsd-current@freebsd.org> Subject: Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75 Message-ID: <20230415160928.6779826E@slippy.cwsent.com> In-Reply-To: <20230415175218.777d0a97@thor.intern.walstatt.dynvpn.de> References: <20230413071032.18BFF31F@slippy.cwsent.com> <D0D9BD06-C321-454C-A038-C55C63E0DD6B@dawidek.net> <20230413063321.60344b1f@cschubert.com> <CAGudoHG3rCx93gyJTmzTBnSe4fQ9=m4mBESWbKVWtAGRxen_4w@mail.gmail.com> <20230413135635.6B62F354@slippy.cwsent.com> <c41f9ed6-e557-9255-5a46-1a22d4b32d66@dawidek.net> <319a267e-3f76-3647-954a-02178c260cea@dawidek.net> <b60807e9-f393-6e6d-3336-042652ddd03c@freebsd.org> <441db213-2abb-b37e-e5b3-481ed3e00f96@dawidek.net> <5ce72375-90db-6d30-9f3b-a741c320b1bf@freebsd.org> <99382FF7-765C-455F-A082-C47DB4D5E2C1@yahoo.com> <32cad878-726c-4562-0971-20d5049c28ad@freebsd.org> <ABC9F3DB-289E-455E-AF43-B3C13525CB2C@yahoo.com> <20230415115452.08911bb7@thor.intern.walstatt.dynvpn.de> <20230415143625.99388387@slippy.cwsent.com> <20230415175218.777d0a97@thor.intern.walstatt.dynvpn.de>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <20230415175218.777d0a97@thor.intern.walstatt.dynvpn.de>, FreeBSD Us er writes: > Am Sat, 15 Apr 2023 07:36:25 -0700 > Cy Schubert <Cy.Schubert@cschubert.com> schrieb: > > > In message <20230415115452.08911bb7@thor.intern.walstatt.dynvpn.de>, > > FreeBSD Us > > er writes: > > > Am Thu, 13 Apr 2023 22:18:04 -0700 > > > Mark Millard <marklmi@yahoo.com> schrieb: > > > > > > > On Apr 13, 2023, at 21:44, Charlie Li <vishwin@freebsd.org> wrote: > > > > > > > > > Mark Millard wrote: > > > > >> FYI: in my original report for a context that has never had > > > > >> block_cloning enabled, I reported BOTH missing files and > > > > >> file content corruption in the poudriere-devel bulk build > > > > >> testing. This predates: > > > > >> https://people.freebsd.org/~pjd/patches/brt_revert.patch > > > > >> but had the changes from: > > > > >> https://github.com/openzfs/zfs/pull/14739/files > > > > >> The files were missing from packages installed to be used > > > > >> during a port's build. No other types of examples of missing > > > > >> files happened. (But only 11 ports failed.) > > > > > I also don't have block_cloning enabled. "Missing files" prior to brt > _rev > > > ert may actually > > > > > be present, but as the corruption also messes with the file(1) signat > ure, > > > some tools like > > > > > ldconfig report them as missing. > > > > > > > > For reference, the specific messages that were not explicit > > > > null-byte complaints were (some shown with a little context): > > > > > > > > > > > > ===> py39-lxml-4.9.2 depends on shared library: libxml2.so - not foun > d > > > > ===> Installing existing package /packages/All/libxml2-2.10.3_1.pkg > > > > > [CA72_ZFS] Installing libxml2-2.10.3_1... > > > > [CA72_ZFS] Extracting libxml2-2.10.3_1: .......... done > > > > ===> py39-lxml-4.9.2 depends on shared library: libxml2.so - found > > > > (/usr/local/lib/libxml2.so) . . . > > > > [CA72_ZFS] Extracting libxslt-1.1.37: .......... done > > > > ===> py39-lxml-4.9.2 depends on shared library: libxslt.so - found > > > > (/usr/local/lib/libxslt.so) ===> Returning to build of py39-lxml-4.9. > 2 > > > > . . . > > > > ===> Configuring for py39-lxml-4.9.2 > > > > Building lxml version 4.9.2. > > > > Building with Cython 0.29.33. > > > > Error: Please make sure the libxml2 and libxslt development packages ar > e in > > > stalled. > > > > > > > > > > > > [CA72_ZFS] Extracting libunistring-1.1: .......... done > > > > ===> libidn2-2.3.4 depends on shared library: libunistring.so - not f > ound > > > > > > > > > > > > > > > [CA72_ZFS] Extracting gmp-6.2.1: .......... done > > > > ===> mpfr-4.2.0,1 depends on shared library: libgmp.so - not found > > > > > > > > > > > > > ===> nettle-3.8.1 depends on shared library: libgmp.so - not found > > > > ===> Installing existing package /packages/All/gmp-6.2.1.pkg > > > > [CA72_ZFS] Installing gmp-6.2.1... > > > > the most recent version of gmp-6.2.1 is already installed > > > > ===> nettle-3.8.1 depends on shared library: libgmp.so - not found > > > > > *** Error code 1 > > > > > > > > > > > > autom4te: error: need GNU m4 1.4 or later: /usr/local/bin/gm4 > > > > > > > > > > > > checking for GNU > > > > M4 that supports accurate traces... configure: error: no acceptable m4 > coul > > > d be found in > > > > $PATH. GNU M4 1.4.6 or later is required; 1.4.16 or newer is recommende > d. > > > > GNU M4 1.4.15 uses a buggy replacement strstr on some systems. > > > > Glibc 2.9 - 2.12 and GNU M4 1.4.11 - 1.4.15 have another strstr bug. > > > > > > > > > > > > ld: error: /usr/local/lib/libblkid.a: unknown file type > > > > > > > > > > > > === > > > > Mark Millard > > > > marklmi at yahoo.com > > > > > > > > > > > > > > Hello > > > > > > whar is the recent status of fixing/mitigate this desatrous bug? Especial > ly f > > > or those with the > > > new option enabled on ZFS pools. Any advice? > > > > > > In an act of precausion (or call it panic) I shutdown several servers to > prev > > > ent irreversible > > > damages to databases and data storages. We face on one host with /usr/por > ts r > > > esiding on ZFS > > > always errors on the same files created while staging (using portmaster, > leav > > > es the system > > > with noninstalled software, i.e. www/apache24 in our case). Deleting the > work > > > folder doesn't > > > seem to change anything, even when starting a scrubbing of the entire poo > l (R > > > AIDZ1 pool) - > > > cause unknown, why it affects always the same files to be corrupted. Same > wit > > > h deve/ruby-gems. > > > > > > Poudriere has been shutdown for the time being to avoid further issues. > > > > > > Are there any advies to proceed apart from conserving the boxes via shutd > own? > > > > > > Thank you ;-) > > > oh > > > > > > > > > > > > -- > > > O. Hartmann > > > > With an up-to-date tree + pjd@'s "Fix data corruption when cloning embedded > > > blocks. #14739" patch I didn't have any issues, except for email messages > > with corruption in my sent directory, nowhere else. I'm still investigating > > > the email messages issue. IMO one is generally safe to run poudriere on the > > > latest ZFS with the additional patch. > > > > My tests of the additional patch concluded that it resolved my last > > problems, except for the sent email problem I'm still investigating. I'm > > sure there's a simple explanation for it, i.e. the email thread was > > corrupted by the EXDEV regression which cannot be fixed by anything, even > > reverting to the previous ZFS -- the data in those files will remain > > damaged regardless. > > > > I cannot speak to the others who have had poudriere and other issues. I > > never had any problems with poudriere on top of the new ZFS. > > > > WRT reverting block_cloning pools to without, your only option is to backup > > > your pool and recreate it without block_cloning. Then restore your data. > > > > > > All right, I interpret the answer that way, that I need a most recent source > tree (and > accordingly built and installed OS) AND a patch that isn't officially commite > d? Yes. > > On a box I'm with: > > FreeBSD 14.0-CURRENT #8 main-n262175-5ee1c90e50ce: Sat Apr 15 07:57:16 CEST 2 > 023 amd64 > > The box is crashing while trying to update ports with the well known issue: > > Panic String: VERIFY(!zil_replaying(zilog, tx)) I never had that problem because I didn't enable block_cloning, except on one pool, which I created a zpool checkpoint to allow for simple rollback if needed. That pool, when tested, had all the patches applied when block_cloning was enabled. It never experienced the panic. > > At the moment all alarm bells are ringing and I lost track about what has bee > n patched and > already commited and what is still as patch available but in the phase of eva > luation or > inofficially emmited here. I maintain a separate branch which contains my own patches on top of what is main. I committed the additional patches on top of it, now a new branch, to keep track of this issue. Maintaining a separate branch saves a lot of grief. I can't help you with this. > > According to the EXDEV issue: in cases of poudriere or ports trees on ZFS, wh > at do I have to > do to ensure that those datasets are clean? The OS should detect file corrupt > ion but in my > case the box is crashing :-( File corruption will not cause a crash. Corruption of metadata might. AFAICT metadata was never corrupted. Only file contents were. > > I did several times scrubbing, but this seems to be the action of a helpless > and desperate man > ... ;-/ Scrubbing doesn't fix file contents that were corrupted prior to the checksum being calculated and written to disk. As far as ZFS is concerned the data is correct. > > Greetings > > Oliver > > -- > O. Hartmann I have for the time being reverted the ZFS import in my day-to-day branch. My branch with the new ZFS + pjd@ patch is currently building. I won't be able to test it until Monday -- because of life things that have taken precedence this weekend. The plan is, starting Monday, to install the branch on my sandbox machine and give it a thorough workout there. Once satisfied I will install it elsewhere. I will report my findings next week. I think it's a good policy for people who track -CURRENT, if they apply patches, commit them to a branch (not main), building and installing from that branch. This avoids all kinds of confusion. This policy will allow you to create your own personal patches you don't wish to share with others. My day-to-day branch contains at a minimum seven patches I never plan on committing. -- Cheers, Cy Schubert <Cy.Schubert@cschubert.com> FreeBSD UNIX: <cy@FreeBSD.org> Web: https://FreeBSD.org NTP: <cy@nwtime.org> Web: https://nwtime.org e^(i*pi)+1=0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20230415160928.6779826E>