Date: Sat, 13 Jun 2020 10:24:49 +1000 From: Dewayne Geraghty <dewayne@heuristicsystems.com.au> To: FreeBSD Stable Mailing List <freebsd-stable@freebsd.org> Subject: Upgrading to 12.1S 362003 - a few issues Message-ID: <978b8cb5-de88-4264-25ec-e7ecaaf80c75@heuristicsystems.com.au>
next in thread | raw e-mail | index | archive | help
After upgrading to 12.1Stable as of June 11:
1) squid - fails with segmentation fault, ldd "Cannot load PIE binary"
2) gcc9 - suffers a cc1 internal compiler error
3) pkg-static - issues "failed" messages, unable to package or install
Environment Xeon E3, ufs2 only, previously running FreeBSD 12.1 dated
1st May (from kernel.old).
Prior to the upgrade all ports were rebuilt without issue, but NOT
installed as they were a fall-back, in the event that clang 10 caused
issues (the concern). There are multiple jails on this system, both
amd64 and i386 - some for building, testing and production use. One of
the production i386 jails runs squid, unchanged since Sept 2019.
/etc/src.conf contains
WITH_PIE=YES
WITH_BIND_NOW=YES
Most of our 1400+ ports are built and run with relro, now, pie and where
possible with noexecstack &/or no-common. These functioned in an ASLR
environment. (ASLR is only disabled during builds (gcc9 complains), or
when there's a problem, now).
Note: NONE of the ports were rebuilt after the upgrade. However as part
of resolution, beep and squid were rebuilt.
=== Sequence of thigns ===
Upgrade performed. System rebooted without incident to
FreeBSD 12.1-STABLE #0 r362003M: Thu Jun 11 23:07:00 AEST 2020 i386
hqdev-amd64-smp-vga 1201517 1201517
but some port/application failures:
Problem 1
---------
i386 jail demonstrated:
# /usr/local/etc/rc.d/squid start
Starting squid.
Segmentation fault
# ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
ldd: /usr/local/sbin/squid: Cannot load PIE binary /usr/local/sbin/squid
as DSO
/usr/local/sbin/squid: exit status 1
Lets check some relevant sysctls:
~# sysctl kern.elf64
kern.elf64.aslr.stack_gap: 0
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 0
kern.elf64.pie_base: 16912384
kern.elf64.nxstack: 0
kern.elf64.fallback_brand: -1
~# sysctl kern.elf32
kern.elf32.aslr.stack_gap: 0
kern.elf32.aslr.honor_sbrk: 1
kern.elf32.aslr.pie_enable: 0
kern.elf32.aslr.enable: 0
kern.elf32.pie_base: 16781312
kern.elf32.read_exec: 0
kern.elf32.nxstack: 0
kern.elf32.fallback_brand: -1
Perhaps this may be helpful to the reader?
# readelf -d /usr/local/sbin/squid
Dynamic section at offset 0x5ddddc contains 39 entries:
Tag Type Name/Value
0x00000001 NEEDED Shared library: [librt.so.1]
0x00000001 NEEDED Shared library: [libcrypt.so.5]
0x00000001 NEEDED Shared library: [libregex.so.1]
0x00000001 NEEDED Shared library: [libcrypto.so.11]
0x00000001 NEEDED Shared library: [libssl.so.11]
0x00000001 NEEDED Shared library: [libm.so.5]
0x00000001 NEEDED Shared library: [libpcreposix.so.0]
0x00000001 NEEDED Shared library: [libpcre.so.1]
0x00000001 NEEDED Shared library: [libkrb5.so.26]
0x00000001 NEEDED Shared library: [libgssapi.so.3]
0x00000001 NEEDED Shared library: [libc++.so.1]
0x00000001 NEEDED Shared library: [libcxxrt.so.1]
0x00000001 NEEDED Shared library: [libgcc_s.so.1]
0x00000001 NEEDED Shared library: [libthr.so.3]
0x00000001 NEEDED Shared library: [libc.so.7]
0x0000001d RUNPATH Library runpath:
[/usr/local/lib/heimdal:/usr/lib:/usr/local/lib]
0x0000000c INIT 0xfe0d8
0x0000000d FINI 0x45e0b0
0x00000019 INIT_ARRAY
0x0000001b INIT_ARRAYSZ 292 (bytes)
0x00000004 HASH 0x19c
0x00000005 STRTAB 0x4a520
0x00000006 SYMTAB 0x155c0
0x0000000a STRSZ 580827 (bytes)
0x0000000b SYMENT 16 (bytes)
0x00000015 DEBUG 0x0
0x00000003 PLTGOT 0x5e0c94
0x00000002 PLTRELSZ 4432 (bytes)
0x00000014 PLTREL REL
0x00000017 JMPREL 0xfcf88
0x00000011 REL 0xded48
0x00000012 RELSZ 123456 (bytes)
0x00000013 RELENT 8 (bytes)
0x6ffffffb FLAGS_1 unknown (0x8000000)
0x6ffffffe VERNEED 0xdebe8
0x6fffffff VERNEEDNUM 9
0x6ffffff0 VERSYM 0xd81fc
0x6ffffffa RELCOUNT 13575
0x00000000 NULL 0x0
Finding the "Cannont load PIE" string in "/usr/src/libexec/rtld-elf/rtld.c"
So commented out the code:
if (obj->z_pie) {
_rtld_error("Cannot load PIE binary %s as DSO", obj->path);
goto errp;
just to get going...
# cd /usr/src/libexec/rtld-elf
# make -DUSE_K8 clean && make -DUSE_K8 && make -DUSE_K8 install
...
cc -O2 -pipe -g0 -ggdb0 -DSTRIP_FBSDID -UDEBUGGING -UDEBUG
-DUSB_HAVE_DISABLE_ENUM -O2 -fno-math-errno -fomit-frame-pointer
-Wno-error=unused-command-line-argument -Wl,--hash-style=sysv
-fno-common -march=haswell -Wall -DFREEBSD_ELF -DIN_RTLD -ffreestanding
-I/smallblocks/src/lib/csu/common
-I/smallblocks/src/libexec/rtld-elf/amd64
-I/smallblocks/src/libexec/rtld-elf -fpic -DPIC -fvisibility=hidden
-mno-mmx -mno-sse -mno-avx -mno-avx2 -msoft-float -DNDEBUG -std=gnu99
-Wsystem-headers -Wall -Wno-format-y2k -W -Wno-unused-parameter
-Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type
-Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter
-Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls
-Wold-style-definition -Wno-pointer-sign -Wformat=2
-Wno-format-extra-args -Wmissing-variable-declarations -Wthread-safety
-Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable
-Qunused-arguments -nostdlib -e .rtld_start -shared -Wl,-Bsymbolic
-Wl,-z,defs -Wl,--version-script=Version.map -Wl,-znow -o ld-elf.so.1
rtld_start.o reloc.o rtld.o rtld_lock.o rtld_malloc.o rtld_printf.o
map_object.o xmalloc.o debug.o libmap.o
-L/usr/obj/smallblocks/src/amd64.amd64/lib/libc -lc_nossp_pic
ld: error: unable to find library -lc_nossp_pic
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1
(Most likely need to build libc - lets skip ahead)
Resort to a previous build (tar file) and pull ONLY /libexec/ld-elf.so.
Installing this, into the i386 jails
-r-xr-xr-x 1 root wheel 134040 20 May 11:39 libexec/ld-elf.so.1
enabled ldd to provide details, the necessary libraries are reachable.
# /usr/local/etc/rc.d/squid start
Starting squid.
Segmentation fault
---
squid also fails in an amd64 jail - segmentation fault.
Problem 2 - gcc 9.3.0 problem cc1 internal compiler error
---------
OK lets try rebuilding the simplest port audio/beep, in an amd64 jail
--- beep.o ---
cc1: internal compiler error: Segmentation fault
libbacktrace could not find executable to open
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://gcc.gnu.org/bugs/>; for instructions.
*** [beep.o] Error code 1
make[1]: stopped in /var/ports/usr/ports/audio/beep/work/beep
1 error
As I force a lot of ports to use gcc9, this was a surprise, especially
as gcc9 is one of the few ports that doesn't use esoteric flags like
pie, bind or noexecstack. Lets use clang 10 instead.
clang 10 builds beep and it runs from
/usr/ports/audio/beep/work/stage/usr/local/bin/beep
PS I use gcc9 due to the availability of -fstack-clash-protection (which
is also in clang 10.1; for another day)
Problem 3 - pkg neither installs nor packages
---------
====> Compressing man pages (compress-man)
===> Building package for beep-1.0_1
pkg-static: failed to get the note section
pkg-static: failed to get the note section
pkg-static: Unable to determine ABI
pkg-static: Cannot parse configuration file!
*** Error code 1
Stop.
This is going to be a problem, as "make -C www/squid clean package"
neither builds a package nor performs an installation. Need some
mechanism to build/install without the pkg infrastructure...
# pkg-static help
pkg-static: failed to get the note section
pkg-static: failed to get the note section
pkg-static: Unable to determine ABI
pkg-static: Cannot parse configuration file!
Rebuilding pkg with clang 10 and copying
/usr/ports/ports-mgmt/pkg/work/stage/usr/local/sbin/pkg-static to
/usr/local/sbin/pkg-static, made no difference.
Back to squid
-------------
OK - lets try a bare bones squid, strip out everything to get a mini-squid
(Modify Makefile - Remove all defaults from squid, except GSSAPI_NONE;
remove perl5 from USES)
Because we can't build a package or install, lets try:
# /usr/ports/www/squid/work/stage/usr/local/sbin/squid -h
YES!!! (remember though, this is the old /libexec/ld-elf.so.1)
The runtime dependencies for "reduced" squid match "fat" squid! Comment
out the refresh_patterns in squid.conf (probably why perl is needed)
And...
squid runs! (Something from Galaxy Quest comes to mind)
Reinstate the original (ie recently updated) ld-elf.so.1. Yep squid
starts and functions properly, though its interesting that
# ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
ldd: /usr/local/sbin/squid: Cannot load PIE binary /usr/local/sbin/squid
as DSO
/usr/local/sbin/squid: exit status 1
---
Conclusion
----------
Some folks, perhaps only those that use pie :
- an upgrade to 12.1S may cause applications to segfault on both i386
and amd64, these applications will require a rebuild
- pkg is problematic, and what in the OS upgrade is causing these
messages (& failure) that weren't there yesterday?
- gcc9 isn't going to be an easy rebuild as it depends on gmake et al,
which won't install...
- iterating through: relro, now, pie, noexecstack indicated no
contribution to the problem. Seems to be something in image
activation... ld-elf ?
Next steps: investigate
failed to get the note -
/usr/ports/ports-mgmt/pkg/work/pkg-1.13.2/libpkg/pkg_elf.c
Unable to determine ABI -
/var/ports/usr/ports/ports-mgmt/pkg/work/pkg-1.13.2/libpkg/pkg_config.c
Cannot parse conf -
/var/ports/usr/ports/ports-mgmt/pkg/work/pkg-1.13.2/src/main.c
PS Re-enabling kern.elf32 and kern.elf64 and restarting the production
applications are functioning happily. Most applications are ok, squid
and gcc9 stand out.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?978b8cb5-de88-4264-25ec-e7ecaaf80c75>