Date: Sat, 19 Mar 2022 18:06:51 +0000 From: Rene Ladan <rene@freebsd.org> To: Mark Johnston <markj@freebsd.org> Cc: Thomas Zander <riggs@freebsd.org>, Matthias Fechner <mfechner@freebsd.org>, Christoph Moench-Tegeder <cmt@burggraben.net>, Bernard Spil <brnrd@freebsd.org>, "ports-committers@FreeBSD.org" <ports-committers@freebsd.org>, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org, secteam@freebsd.org Subject: Re: git: 43741377b143 - main - security/openssl: Security update to 1.1.1n Message-ID: <YjYbupCRBpTUGaR2@freefall.freebsd.org> In-Reply-To: <YjYIEx8GQzZDSLaW@framework> References: <CAFU734wn1kqoQfpuxT%2BSQdZ--VFKawhHwwFimj6Z8RyzLD-bfQ@mail.gmail.com> <YjTV6f5B8IAUFoWm@elch.exwg.net> <YjTb28jXMkJplc8s@elch.exwg.net> <CAFU734zacFy_%2BDvwG6W55X1=Yu8FYLvTV6DbyhW1azkUwXzG3Q@mail.gmail.com> <YjTrVsQi0ZTdadQK@elch.exwg.net> <579f562b-8add-d3f7-77c9-1a6126bd282b@FreeBSD.org> <CAFU734w5ycC1aY4x1Qoy3SL0Og=8osq%2BhG-mBvaLFrLYHm-MOw@mail.gmail.com> <YjW6ZCkdIMq/lXGA@freefall.freebsd.org> <CAFU734zX9DJSsFv=46=6GdVdzxk%2B0yZz-brALdLcdY0m8BQ6Dg@mail.gmail.com> <YjYIEx8GQzZDSLaW@framework>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Mar 19, 2022 at 12:42:59PM -0400, Mark Johnston wrote: > On Sat, Mar 19, 2022 at 12:40:45PM +0100, Thomas Zander wrote: > > On Sat, 19 Mar 2022 at 12:11, Rene Ladan <rene@freebsd.org> wrote: > > > > > > On Sat, Mar 19, 2022 at 11:04:58AM +0100, Thomas Zander wrote: > > > > On Sat, 19 Mar 2022 at 09:00, Matthias Fechner <mfechner@freebsd.org> wrote: > > > > > > > > > I can confirm now, the problem is definitely related to the -p8 update. > > > > > I rolled back now to -p7 using `freebsd-update rollback`. > > > > > [...] > > > > > System is now up and running again. > > > > > This all works even if poudriere jail is using -p8. No need to downgrade the jail/base version poudriere is using. > > > > > It is caused by the kernel so the ZFS patch seems to be broken and -p8 should maybe not rolled out to not break more systems of users. > > > > > > > > On top of "stop rollout", there is the question how to identify the > > > > broken files for the users who have already upgraded to -p8. A `zpool > > > > scrub` presumably won't help. > > > > > > I think it also applies to 13.1-BETA2 ? > > > > > > Should we involve/CC some src committers? > > > > I have just rolled back to -p7 and run a number of test builds in > > poudriere (the jails still have the -p8 user land). I see the same as > > Matthias and Christoph, the rollback to the -p7 kernel/zfs resolved > > the build problems, there are no NUL byte files generated anymore. > > Adding markj@ to the discussion. Mark, the TLDR so far: > > - One of the zfs patches in -p8 seems to cause erroneous writes. > > - We noticed because of many build failures with poudriere (presumably > > highly io-loaded during build). > > - Symptom: Production of files with large runs of NUL-bytes. > I'm not sure if it is the same problem or something that popped up at the same time, but since on a 13.1-BETA2-amd64 host updated with freebsd-update and a poudriere 13.1-BETA2-armv6x jail built from git+https building lang/python38 reliably hangs in the stage phase until poudriere decides to kill it: rene@e17:~ % ps axww | grep python38 52652 8 S+ 0:00.00 grep python38 12503 12 I 0:00.03 sh: poudriere[13_1-armv6x-quarterly][01]: build_pkg (python38-3.8.12_1) (sh) 42562 12 I 0:00.00 sh: poudriere[13_1-armv6x-quarterly][01]: build_pkg (python38-3.8.12_1) (sh) 42563 12 IJ 0:00.02 /usr/bin/make -C /usr/ports/lang/python38 stage 42622 12 IJ 0:00.02 /usr/bin/make -f Makefile INSTALL_SHARED=install -s -m 0644 DESTDIR=/wrkdirs/usr/ports/lang/python38/work/stage altinstall 46066 12 IJ 0:05.91 /usr/local/bin/qemu-arm-static ./python -E -Wi -OO /wrkdirs/usr/ports/lang/python38/work/stage/usr/local/lib/python3.8/compileall.py -j0 -d /usr/local/lib/python3.8 -f -x bad_coding|badsyntax|site-packages|lib2to3/tests/data /wrkdirs/usr/ports/lang/python38/work/stage/usr/local/lib/python3.8 46101 12 IJ 0:04.07 /usr/local/bin/qemu-arm-static ./python -E -Wi -OO /wrkdirs/usr/ports/lang/python38/work/stage/usr/local/lib/python3.8/compileall.py -j0 -d /usr/local/lib/python3.8 -f -x bad_coding|badsyntax|site-packages|lib2to3/tests/data /wrkdirs/usr/ports/lang/python38/work/stage/usr/local/lib/python3.8 46102 12 IJ 0:04.09 /usr/local/bin/qemu-arm-static ./python -E -Wi -OO /wrkdirs/usr/ports/lang/python38/work/stage/usr/local/lib/python3.8/compileall.py -j0 -d /usr/local/lib/python3.8 -f -x bad_coding|badsyntax|site-packages|lib2to3/tests/data /wrkdirs/usr/ports/lang/python38/work/stage/usr/local/lib/python3.8 46104 12 IJ 0:04.28 /usr/local/bin/qemu-arm-static ./python -E -Wi -OO /wrkdirs/usr/ports/lang/python38/work/stage/usr/local/lib/python3.8/compileall.py -j0 -d /usr/local/lib/python3.8 -f -x bad_coding|badsyntax|site-packages|lib2to3/tests/data /wrkdirs/usr/ports/lang/python38/work/stage/usr/local/lib/python3.8 rene@e17:~ % > I've had zero luck reproducing this locally. I built several hundred > ports, including textproc/py-pystemmer mentioned elsewhere in the > thread, without any failures or instances of zero-filled files. Another > member of secteam also hasn't been able to trigger any build failures on > -p8. Any hints on a reproducer would be useful. > I haven't tested anything on 13.0-p8 yet. > We can simply push a -p9 which reverts EN-22:10 and :11, but of course > it would be preferable to precisely identify the problem. Perhaps there are some tests/commands I can run? Regards, René
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YjYbupCRBpTUGaR2>