From nobody Mon Aug 11 00:17:29 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4c0ZwQ3PRBz63djW; Mon, 11 Aug 2025 00:17:58 +0000 (UTC) (envelope-from freebsd@gushi.org) Received: from prime.gushi.org (prime.gushi.org [IPv6:2620:137:6000:10::142]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (secp384r1) client-digest SHA384) (Client CN "prime.gushi.org", Issuer "E6" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4c0ZwN2f6fz3mKG; Mon, 11 Aug 2025 00:17:56 +0000 (UTC) (envelope-from freebsd@gushi.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gushi.org header.s=prime2014 header.b=uMiPa8aE; spf=pass (mx1.freebsd.org: domain of freebsd@gushi.org designates 2620:137:6000:10::142 as permitted sender) smtp.mailfrom=freebsd@gushi.org; dmarc=pass (policy=none) header.from=gushi.org Received: from smtpclient.apple ([IPv6:2001:500:6b:200:8000:0:0:c]) (authenticated bits=0) by prime.gushi.org (8.18.1/8.18.1) with ESMTPSA id 57B0HkNW032181 (version=TLSv1.2 cipher=ECDHE-ECDSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 11 Aug 2025 00:17:47 GMT (envelope-from freebsd@gushi.org) DKIM-Filter: OpenDKIM Filter v2.10.3 prime.gushi.org 57B0HkNW032181 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gushi.org; s=prime2014; t=1754871467; bh=FgkmbxqNqWH2DSXzjSfJYAhXK+6jFVeiY3Tdbq0M2/M=; h=Subject:From:In-Reply-To:Date:Cc:References:To; z=Subject:=20Re:=20HEADS-UP:=20pkgbase=20pkg=20upgrade=20breaks=20s ystems=20Re:=20after=20updating=0D=0A=20to=20latest,=20pkg=20base= 20segfaults=20and=20leaves=20me=20unbootable|From:=20"Dan=20Mahone y=20(Ports)"=20|In-Reply-To:=20<582d802f-da93-4 15c-9dee-1f1f1beb1830@smeets.xyz>|Date:=20Sun,=2010=20Aug=202025=2 017:17:29=20-0700|Cc:=20freebsd-pkgbase@freebsd.org,=0D=0A=20freeb sd-current@freebsd.org|References:=20=0D=0A=20<582d802f-da93-415c-9dee-1f1f1beb1830@ smeets.xyz>|To:=20Florian=20Smeets=20; b=uMiPa8aEl3fpkaT4H+TY3AwtWbnaIE+Y77I5j8rkc0EAaC20VvznjdbnCOqbI19y2 ez2hjVokxRzIPdwVmWKF0JtqcckGnU+m3z5wGWY6oBMCse4WgMu0vcWSWSVokH4L/D eCe45aqRIXuu62j9M9FAcmCmJPNnLY0Xp7Iv0C04jP257ZYYq1fgc+4aDtO49lfgoE 7sj05gIXWUUpgoG4DfOzYs24g9mTeb51NVvIeBCsAB9O242GkBC5UuagvMZbFXuXcd mfu62uX6TKEOb9Xb847xirNlsRo4Ptq+/4fR5TBMy9SFyGLxDqpskSoM4UE2PVyfVy EsxbEtJ66SrUw== X-Authentication-Warning: prime.gushi.org: Host [IPv6:2001:500:6b:200:8000:0:0:c] claimed to be smtpclient.apple Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3863.100.1\)) Subject: Re: HEADS-UP: pkgbase pkg upgrade breaks systems Re: after updating to latest, pkg base segfaults and leaves me unbootable From: "Dan Mahoney (Ports)" In-Reply-To: <582d802f-da93-415c-9dee-1f1f1beb1830@smeets.xyz> Date: Sun, 10 Aug 2025 17:17:29 -0700 Cc: freebsd-pkgbase@freebsd.org, freebsd-current@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <5DCB0B5E-3582-4877-A1B8-894C4EA68F13@gushi.org> References: <582d802f-da93-415c-9dee-1f1f1beb1830@smeets.xyz> To: Florian Smeets X-Mailer: Apple Mail (2.3863.100.1) X-Spamd-Result: default: False [-5.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[gushi.org:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gushi.org,none]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; R_DKIM_ALLOW(-0.20)[gushi.org:s=prime2014]; ONCE_RECEIVED(0.20)[]; RCVD_IN_DNSWL_MED(-0.20)[2620:137:6000:10::142:from]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; APPLE_MAILER_COMMON(0.00)[]; DKIM_TRACE(0.00)[gushi.org:+]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org,freebsd-pkgbase@freebsd.org]; RCPT_COUNT_THREE(0.00)[3]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:393507, ipnet:2620:137:6000::/44, country:US]; TO_DN_SOME(0.00)[]; HAS_XAW(0.00)[] X-Rspamd-Queue-Id: 4c0ZwN2f6fz3mKG X-Spamd-Bar: ----- > On Aug 9, 2025, at 12:55=E2=80=AFAM, Florian Smeets = wrote: >=20 >=20 > FWIW, I revived the system by deleting all newer packages rm = /var/cache/pkg/*snap20250808* and just untaring the stuff in = /var/cache/pkg to / >=20 > cd / ; for i in `/rescue/ls -1 /var/cache/pkg/FreeBSD-* ` ; do = /rescue/tar xvzf $i ; done >=20 > After that, I resorted to building from source and installing to get = the system into a half way consistent state with a chance of surviving a = reboot. So, looking at Florian's solution, I came across two issues. 1) After you build from src, pkg thinks you have one thing installed, = and buildworld has installed something different. I cannot figure out = how to tell pkg "delete anything you know from the pkgdb about these = pkgs, and just let me re-run pkgbasify". There's a -O option that = exists in netbsd's pkg, and I'm sure there's some absurd invocation that = involves manipulating the raw pkgdb, but nothing "easy"/ 2) Pkgbase got me into this problem, and I wanted it to get me out. I = had all the pkgs sitting there in /var/cache/pkg, ready to go, I just = needed to get them in there. Doing a pkg-info yielded a scant 20 = packages or so. Worse still: /rescue/ls /var/cache/pkg/FreeBSD-* gave me "argument list = too long" and with no grep, I basicaly had to rely on scrollback to get = what I was looking for. Ergo, my solution was more like: cd /var/cache /rescue/cp -pR pkg pkg-backup mkdir ../pkg-snap20250808125411 cp FreeBSD*snap20250808125411* ../pkg-snap20250808125411 (=46rom there, I used rm cautiously to remove all the -dbg packages -- I = can put them back later, but for right now I needed them to work. Let's just try one package... # ABI=3DFreeBSD:15:amd64 /usr/local/sbin/pkg-static add -M = ./FreeBSD-libarchive-15.snap20250808125411.pkg pkg-static: Setting ABI requires setting OSVERSION, guessing the = OSVERSION as: 1500000 Installing FreeBSD-libarchive-15.snap20250808125411... Newer FreeBSD version for package FreeBSD-libarchive: To ignore this error set IGNORE_OSVERSION=3Dyes - package: 1500058 - running userland: 1500000 Ignore the mismatch and continue? [y/N]: y `-- Installing FreeBSD-runtime-15.snap20250808125411... | `-- Installing FreeBSD-libarchive-15.snap20250808125411... | | `-- Installing FreeBSD-runtime-15.snap20250808125411... | | `-- Installing FreeBSD-libarchive-15.snap20250808125411... | | | `-- Installing FreeBSD-runtime-15.snap20250808125411... | | | `-- Installing FreeBSD-libarchive-15.snap20250808125411... | | | | `-- Installing FreeBSD-runtime-15.snap20250808125411... | | | | `-- Installing = FreeBSD-libarchive-15.snap20250808125411... ^C (Okay, so there's no "y" option to pkg add... and these things will = forever sit and try to install circular dependencies. Dumb). New solution: What if we're in a new directory that only has packages in = it that have already been installed? mkdir /tmp/foo cd /var/cache/pkg-snap20250808125411/ cp FreeBSD-libarchive-15.snap20250808125411.pkg /tmp/foo cd /tmp/foo ABI=3DFreeBSD:15:amd64 /usr/local/sbin/pkg-static add -M = ./FreeBSD-libarchive-15.snap20250808125411.pkg pkg-static: Setting ABI requires setting OSVERSION, guessing the = OSVERSION as: 1500000 Installing FreeBSD-libarchive-15.snap20250808125411... Newer FreeBSD version for package FreeBSD-libarchive: To ignore this error set IGNORE_OSVERSION=3Dyes - package: 1500058 - running userland: 1500000 Ignore the mismatch and continue? [y/N]: y pkg-static: Missing shlib dependency: libbsdxml.so.4 pkg-static: Missing shlib dependency: libbz2.so.4 pkg-static: Missing shlib dependency: libcrypto.so.30 pkg-static: Missing shlib dependency: liblzma.so.5 pkg-static: Missing shlib dependency: libprivatezstd.so.5 pkg-static: Missing shlib dependency: libz.so.6 Extracting FreeBSD-libarchive-15.snap20250808125411: 100% Okay, but this will still be exhausting if I need to keep hitting = "yes"... for i in `ls -1 /var/cache/pkg-snap20250808125411/` > do > cp /var/cache/pkg-snap20250808125411/$i /tmp/foo > cd /tmp/foo > OSVERSION=3D1500058 ABI=3DFreeBSD:15:amd64 /usr/local/sbin/pkg-static = add -M ./$i > done Installing FreeBSD-acct-15.snap20250808125411... pkg-static: Missing shlib dependency: libxo.so.0 Extracting FreeBSD-acct-15.snap20250808125411: 100% Installing FreeBSD-acct-man-15.snap20250808125411... Extracting FreeBSD-acct-man-15.snap20250808125411: 100% Installing FreeBSD-acpi-15.snap20250808125411... [...] Installing FreeBSD-zfs-15.snap20250808125411... Extracting FreeBSD-zfs-15.snap20250808125411: 100% Installing FreeBSD-zfs-dev-15.snap20250808125411... Extracting FreeBSD-zfs-dev-15.snap20250808125411: 100% Installing FreeBSD-zfs-dev-lib32-15.snap20250808125411... pkg-static: Missing shlib dependency: libicp.so.3:32 pkg-static: Missing shlib dependency: libzfsbootenv.so.1:32 Extracting FreeBSD-zfs-dev-lib32-15.snap20250808125411: 100% Installing FreeBSD-zfs-lib32-15.snap20250808125411... Extracting FreeBSD-zfs-lib32-15.snap20250808125411: 100% Installing FreeBSD-zfs-man-15.snap20250808125411... Extracting FreeBSD-zfs-man-15.snap20250808125411: 100% Installing FreeBSD-zoneinfo-15.snap20250808125411... ...I started a normal shell (/bin/csh) and tried a service command, and = sshd restarted. But pkg is still being persnickety, and I don't know what the mechanism = is to bootstrap it is: # ldd /usr/local/sbin/pkg /usr/local/sbin/pkg: libelf.so.2 =3D> /lib/libelf.so.2 (0x2a08ac8d1000) libjail.so.1 =3D> /lib/libjail.so.1 (0x2a08ad283000) libm.so.5 =3D> /lib/libm.so.5 (0x2a08adcbd000) libarchive.so.7 =3D> /usr/lib/libarchive.so.7 (0x2a08ae37a000) libz.so.6 =3D> /lib/libz.so.6 (0x2a08abe01000) libutil.so.9 =3D> not found (0) libssl.so.30 =3D> /usr/lib/libssl.so.30 (0x2a08af3bb000) libcrypto.so.30 =3D> /lib/libcrypto.so.30 (0x2a08af8a2000) libthr.so.3 =3D> /lib/libthr.so.3 (0x2a08b0575000) libc.so.7 =3D> /lib/libc.so.7 (0x2a08b19a8000) libbz2.so.4 =3D> /usr/lib/libbz2.so.4 (0x2a08b257a000) liblzma.so.5 =3D> /usr/lib/liblzma.so.5 (0x2a08b2d8e000) libbsdxml.so.4 =3D> /lib/libbsdxml.so.4 (0x2a08b3302000) libprivatezstd.so.5 =3D> /usr/lib/libprivatezstd.so.5 = (0x2a08b3d33000) libsys.so.7 =3D> /lib/libsys.so.7 (0x2a08b4a90000) libmd.so.7 =3D> /lib/libmd.so.7 (0x2a08b4f76000) [vdso] (0x2a08ab287000) I've got a libutil.so.10, but not 9. Doing a pkg-static install -f pkg = doesn't update it to use .so.10 either. Where does one get this file? =3D=3D=3D=3D I don't have another system with pkgbase on it, so I can't ask what pkg = provides that file. I don't know if, given a list of .pkg files, = there's a way to poll them all and ask. (pkg grep?) While this is still a prerelease thing -- these recipes should be in the = pkgbase wiki page. Knowing how to recover from a pkgbase failure -- or = how to rectify a world and pkgbase that are out of step with each other, = are going to be things we're going to need to do occasionally. >=20 > When starting my upgrade I saw that it wanted to remove a lot of non = pkgbase packages (how are we doing to differentiate pkgbase packages and = "ports" packages in the future?). I thought this might be related to the = krb5 thing, so I created an up to date poudriere jail via pkgbase method = and rebuilt all my pkgs, but even then I saw the same thing as Den, that = pkg wanted to remove a lot of ports pkgs, as this system is not = important I thought I can resolve that after the pkgbase upgrade and = started the upgrade. >=20 > I didn't save scroll back. In my case I saw at the top the first ~100 = pkg transactions were uninstalling pkgbase pkgs, then it upgraded some, = then pkg exited with a segfault. >=20 > Leaving me with ELF interpreter /libexec/ld-elf.so.1 not found, error = 2 >=20 > One thing I checked was /libexec/ was completely empty. >=20 > Florian >=20 >> root@poudriere:/home/dmahoney # pkg-static install -f pkg >> pkg-static: Unable to determine the ABI, none of the ABI_FILEs can be = read. >> pkg-static: Cannot parse configuration file! >> root@poudriere:/home/dmahoney # /rescue/sh >> Cannot read termcap database; >> using dumb terminal settings. >> # pkg-static >> pkg: not enough arguments >> Usage: pkg [-v] [-d] [-l] [-N] [-j |-c |-r ] [-C ] [-R ] [-o = var=3Dvalue] [-4|-6] [] >> For more information on available commands and options see 'pkg = help'. >> # pkg-static install -f pkg >> pkg-static: Unable to determine the ABI, none of the ABI_FILEs can be = read. >> Now, if this were 14.x, I'd fix this by untarring a distfile right = over / to get back up and running. Not quite an option in 15, is it? = (I would encourage the people making this stuff to please consider = keeping those built, even if bsdinstall uses pkgbase). >> This is dayjob's poudriere system -- the build process for it is well = documented and the parts of it that aren't managed by puppet are easily = managed because I captured every command used to create every jail and = ports tree (which was slow because many of them came from = freebsd-archive; both old jails and old copies of ports trees last known = to work with given versions of FreeBSD (kind of required when you needed = to build packages in 2020 because remote hands were Not An Option). = Like all our systems, the bits we care about (homedirs, /usr/local/etc) = are in backups as well. We kind of need it to work. >> So this isn't an emergency. This system is really there so that me, = as a port maintainer, can build a debug build of something, but this = machine isn't in our critical path. ...but What If It Was? We run = Critical Stuff, out there on lone servers in faraway places (on bare = metal) >> But it's *really* not instilling me with a lot of confidence in the = readiness of this pkgbase idea. (For the record, I've also had = freebsd-update leave me dead on the table in similar ways in the past. = I literally called them out during a BSDcan talk without trying to bash = Colin too hard). >> I can capture more scrollback if people want, but I wasn't doing any = of the crazy -f commands people are talking about. This was literally a = "pkg upgrade". >> Full command output is over at = https://users.isc.org/~dmahoney/failedupgrade.txt if devs want to have a = look and try to black-box it. I'll keep the VM running (and logged in, = in a screen session) if there are things people want me to try. >> This begs the questions: >> * Is there CI for pkgbase that tries to upgrade from whatever version = is immediately previous to it, before publishing it? (I know that's = what version I was running, it's the version everyone's been stuck at = for weeks!) >> * For all the debate about "pkgbase and pkg should be exactly the = same", perhaps pkgbase could have an auto-bectl in it? >> * Is there support somewhere for having a lockstep "set" of packages = that one knows are in /var/db/pkg and constitute a relatively concurrent = install? (Or a list of files that I could tell pkg-static to install = with a glob, out of /var/cache/pkg) >> * With something like -CURRENT, is there any support for saying "Okay = not the current-current tree, but current-minus-one" (I guess that = would be if you're working with the weekly builds, but that's not quite = the same. Your only option is "base system of the now" but it happens = less frequently). >> * And for running -CURRENT where this kind of breakage can happen, = could we get a statically linked version of pkg? >> Any questions, let me know. >> -Dan >> Sent from my iPhone >=20 >