Date: Mon, 4 Sep 2023 17:16:56 -0700 From: Mark Millard <marklmi@yahoo.com> To: Current FreeBSD <freebsd-current@freebsd.org>, FreeBSD Mailing List <freebsd-ports@freebsd.org> Cc: "jilles@freebsd.org" <jilles@FreeBSD.org>, Bryan Drewery <bdrewery@FreeBSD.org> Subject: FYI: ^T use during poudriere bulk vs. /bin/sh operation: I got a "Unsafe ckmalloc() call" Abort trap that left a mess Message-ID: <6B7D2081-B152-400F-A294-11F039077A9F@yahoo.com> References: <6B7D2081-B152-400F-A294-11F039077A9F.ref@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
During a (zfs based) poudriere bulk -a run a ^T got a: Unsafe ckmalloc() call Abort trap (core dumped) from /bin/sh . AT that point it had built something like 350 ports into packages in .building/ . After the last finish was reported for the already building ports, I killed 2 stuck poudriere related processes (the rest were gone already) and then I used: # poudriere jail -jmain-amd64-bulk_a -k [00:06:03] Unmounting file systems umount: unmount of = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/tests = failed: Device busy umount: unmount of = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/share = failed: Device busy umount: unmount of = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/lib32 = failed: Device busy umount: unmount of = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/rescue = failed: Device busy umount: unmount of = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/tests = failed: Device busy umount: unmount of = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/share = failed: Device busy umount: unmount of = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/lib32 = failed: Device busy umount: unmount of = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/rescue = failed: Device busy chflags: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/rescue/rcorder:= Read-only file system chflags: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/rescue/mdconfig= : Read-only file system chflags: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/rescue/mdmfs: = Read-only file system . . . chflags: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/tests/lib: = Read-only file system chflags: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/tests: = Read-only file system rm: /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/rescue: = Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/lib32: = Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/share: = Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr/tests: = Device busy rm: /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/usr: = Directory not empty rm: /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/dev: = Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/var/db/po= rts: Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/var/db: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/var/empty= : Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/var: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/rescue: = Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/sbin/init= : Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/sbin: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/libexec/l= d-elf.so.1: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/libexec/l= d-elf32.so.1: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/libexec: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/packages:= Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/proc: = Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/distfiles= : Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/compat/li= nux/proc: Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/compat/li= nux: Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/compat: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/lib/libth= r.so.3: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/lib/libc.= so.7: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/lib/libcr= ypt.so.5: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/lib: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/src: = Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/bin/c= rontab: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/bin/p= asswd: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/bin/c= hpass: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/bin/s= u: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/bin/l= ogin: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/bin: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/tests= : Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/ports= : Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/lib/l= ibrt.so.1: Operation not permitted rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/lib: = Directory not empty rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/lib32= : Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr/share= : Device busy rm: = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40/usr: = Directory not empty rm: /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../40: = Directory not empty . . . rm: /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/ref/../: = Invalid argument After that "df -m" shows lots of "Mounted on"s matching the patterns: /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/rescue /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/usr/lib32 /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/usr/share /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/usr/tests /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/var/db/ports /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/packages /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/usr/src /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/usr/ports /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/distfiles /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/dev /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/dev/fd = /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/compat/linux/proc= /usr/local/poudriere/data/.m/main-amd64-bulk_a-default/*/proc No clue if I'll ever reproduce it. The odd part of the context was I was testing bulk -J128 (no ALLOW_MAKE_JOBS use) on a ThreadRipper 1950X, so 32 hardware threads. This was part of attempting to see if I get any examples of deadlocks or corruptions or such that some earlier imports of openzfs had produced for others. For reference: # uname -apKU FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #118 = main-n265152-f49d6f583e9d-dirty: Mon Sep 4 14:26:56 PDT 2023 = root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG amd64 amd64 1500000 1500000 My attribution to ^T handling is unverified: I did not find the sh.core file. It is just what the timing looked like. =3D=3D=3D Mark Millard marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6B7D2081-B152-400F-A294-11F039077A9F>