From owner-freebsd-stable@freebsd.org Sat Jun 13 00:25:54 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4130A34AEDF for ; Sat, 13 Jun 2020 00:25:54 +0000 (UTC) (envelope-from dewayne@heuristicsystems.com.au) Received: from hermes.heuristicsystems.com.au (hermes.heuristicsystems.com.au [203.41.22.115]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2560 bits) client-digest SHA256) (Client CN "hermes.heuristicsystems.com.au", Issuer "Heuristic Systems Type 4 Host CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 49kJKW3DYhz4HpL for ; Sat, 13 Jun 2020 00:25:50 +0000 (UTC) (envelope-from dewayne@heuristicsystems.com.au) Received: from [10.0.5.3] (noddy.hs [10.0.5.3]) (authenticated bits=0) by hermes.heuristicsystems.com.au (8.15.2/8.15.2) with ESMTPSA id 05D0PKxH064364 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT) for ; Sat, 13 Jun 2020 10:25:20 +1000 (AEST) (envelope-from dewayne@heuristicsystems.com.au) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=heuristicsystems.com.au; s=hsa; t=1592007920; x=1592612721; bh=WAImy6j1QA+oEALibFhOJPQeMbgaBVbdRFviEIDcpp4=; h=From:To:Subject:Message-ID:Date; b=LzXxQEZGflEMJCxTJIFEm1kSGftmfJiDj6RHs3FN4nogix5yUd5yiPcatVbanX6ks 2WZJfAf/FsGacegZ3ci5b6kwkpYWzor3rTKQ4MLs5C9YAyGZY8vIfcsp8t3qxgG9Zm sKjFF6PboehMEqbmvqs9eMZyBC/olBQO1Z4AStIuzFEsNTCFgrCAd X-Authentication-Warning: b3.hs: Host noddy.hs [10.0.5.3] claimed to be [10.0.5.3] From: Dewayne Geraghty Autocrypt: addr=dewayne@heuristicsystems.com.au; prefer-encrypt=mutual; keydata= mQFNBFbOsVMBCgDfvi2PspSwoMEtFhF+aFLQKtzSA9f0dhDqthKHESdfbqxvKzhkBjvTJ5Na EgjKoKfoQTh5xuIv3HLhtDo5PeasPgQl9cPJeriqmqlS+UhY5BGYcMc1AO/TX0fsDaQz96ko at3RUW7sff/qPgVzSurk+DV5h866gPdn5Jdjohyl2F1rzRl6dnaAIyg49zlwZOnPHJGKye+B meqUCnPRglhkpNqXR3v1ulbWpfwhdNDvWT82qTG/qsFy/agjJvxwLuEBeoGc1dPWasO8Nztt 0dqf1Lpeg6SX2yJd76WVS4znt88OEbx/QL2PTJ/YtSepS68WaeKuARKPukkU+QXDep0gaLPl /TvU5xAZndNB3rYnpmoLb32pDHlrJbZUVyTMqc3J2EYM6aaizCpg4VEvVpVSqUT4D9MuREhu PeZ3SvEazQARAQABiQF3BB8BCAAhBQJWzrFTFwyAAWHe5yZt8RJL0vaU1MfDto5dBmeFAgcA AAoJEJVk7a1LmFrdy2QJ/AysDdFIMCRiaqEellprZQyEz5I/qZJEi6yRfXH813hhISFz6moh urZYLQ9SRdyMntT8W3Oc4pJc9fF9RSnY0SSQY/arZbrvsv6hKb1KtIK7P5mLS914J9buxEcJ SWeVuOuMA9aCNqg5uMu19pH5pXayORfbv+K7vFPiyllZ64ShUWZJL69vAc/TsbvMrGtG1M4P qyWCOKEiUT93zhVGQoA0aUYjMAZoyvozZCuieo4O8hkPgMz9lka+3bqQBSOB+qO4Iz+CZs0k Lw7Soga6bRqLK86DH99WjTA6Oj1r8Won+j4V9fnTDCVJoSyqdVHLySDv/lHaNu4Ia4AO4i2d shmLw03gOUvoWLJx5X01A5Zio4FvecnpZqQ0Wz5Ph9MiK3lwarfjonTOLeNGd5BpdnHu5VRC fJml7uAYeyKsD8C4tEBEZXdheW5lIEdlcmFnaHR5IDxkZXdheW5lLmdlcmFnaHR5QGNvbnNj aXVtaW50ZXJuYXRpb25hbC5jb20uYXU+iQGXBBMBCABBAhshCwsKDQkIDAcLAwIECBUKCQgL AwIBBRYDAgEAAh4BAheAFiEEC8bIxjMx+sDl4ZCClWTtrUuYWt0FAl5UUOgACgkQlWTtrUuY Wt3xZAn/W/mq5nDhLIfqxVM9GbU8rGzNsGLfnt5NCVcWlBKhgxOOw9EWkcRTMymwX9OMqwxI +te6Gvy7rG53T2xprtsQyqESZmjWcUSEPsQ9hjw4VZCL15ftBeZMYyO2T1e41UImXAlftleT 2kXCktgyAfwfCzHhFiZM8k9QMFQV1x+JukJ9xPFBgICRLsLsVNVw/R1L7KqARuws4HqXxY1J SCpO+FB4b6tWSIRKbzlb6tctdKppKbG/adVYuoK61ngvmsAzy/9OLhF8u1MNCgyFd2woOErh /zyuap8KvJZMlwAIqpjsoHyXsa0cq8A/uNQSmodwBpRsEGXCmZIZq2FJw6N+38to8C8m97q0 YWrY63VsoA6hA4A4/ywzE3EiwGvqJQBMRv2ET3TIdTyLoEIwXq2bDPU7XTZGh5UZEsKFMHH5 228= To: FreeBSD Stable Mailing List Subject: Upgrading to 12.1S 362003 - a few issues Message-ID: <978b8cb5-de88-4264-25ec-e7ecaaf80c75@heuristicsystems.com.au> Date: Sat, 13 Jun 2020 10:24:49 +1000 User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49kJKW3DYhz4HpL X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=heuristicsystems.com.au header.s=hsa header.b=LzXxQEZG; dmarc=none; spf=pass (mx1.freebsd.org: domain of dewayne@heuristicsystems.com.au designates 203.41.22.115 as permitted sender) smtp.mailfrom=dewayne@heuristicsystems.com.au X-Spamd-Result: default: False [-5.48 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[heuristicsystems.com.au:s=hsa]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_MED(-2.00)[heuristicsystems.com.au:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.02)[-1.017]; DMARC_NA(0.00)[heuristicsystems.com.au]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[203.41.22.115:from]; DKIM_TRACE(0.00)[heuristicsystems.com.au:+]; NEURAL_HAM_SHORT(-0.25)[-0.250]; NEURAL_HAM_MEDIUM(-1.01)[-1.013]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1221, ipnet:203.40.0.0/13, country:AU]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jun 2020 00:25:54 -0000 After upgrading to 12.1Stable as of June 11: 1) squid - fails with segmentation fault, ldd "Cannot load PIE binary" 2) gcc9 - suffers a cc1 internal compiler error 3) pkg-static - issues "failed" messages, unable to package or install Environment Xeon E3, ufs2 only, previously running FreeBSD 12.1 dated 1st May (from kernel.old). Prior to the upgrade all ports were rebuilt without issue, but NOT installed as they were a fall-back, in the event that clang 10 caused issues (the concern). There are multiple jails on this system, both amd64 and i386 - some for building, testing and production use. One of the production i386 jails runs squid, unchanged since Sept 2019. /etc/src.conf contains WITH_PIE=YES WITH_BIND_NOW=YES Most of our 1400+ ports are built and run with relro, now, pie and where possible with noexecstack &/or no-common. These functioned in an ASLR environment. (ASLR is only disabled during builds (gcc9 complains), or when there's a problem, now). Note: NONE of the ports were rebuilt after the upgrade. However as part of resolution, beep and squid were rebuilt. === Sequence of thigns === Upgrade performed. System rebooted without incident to FreeBSD 12.1-STABLE #0 r362003M: Thu Jun 11 23:07:00 AEST 2020 i386 hqdev-amd64-smp-vga 1201517 1201517 but some port/application failures: Problem 1 --------- i386 jail demonstrated: # /usr/local/etc/rc.d/squid start Starting squid. Segmentation fault # ldd /usr/local/sbin/squid /usr/local/sbin/squid: ldd: /usr/local/sbin/squid: Cannot load PIE binary /usr/local/sbin/squid as DSO /usr/local/sbin/squid: exit status 1 Lets check some relevant sysctls: ~# sysctl kern.elf64 kern.elf64.aslr.stack_gap: 0 kern.elf64.aslr.honor_sbrk: 1 kern.elf64.aslr.pie_enable: 0 kern.elf64.aslr.enable: 0 kern.elf64.pie_base: 16912384 kern.elf64.nxstack: 0 kern.elf64.fallback_brand: -1 ~# sysctl kern.elf32 kern.elf32.aslr.stack_gap: 0 kern.elf32.aslr.honor_sbrk: 1 kern.elf32.aslr.pie_enable: 0 kern.elf32.aslr.enable: 0 kern.elf32.pie_base: 16781312 kern.elf32.read_exec: 0 kern.elf32.nxstack: 0 kern.elf32.fallback_brand: -1 Perhaps this may be helpful to the reader? # readelf -d /usr/local/sbin/squid Dynamic section at offset 0x5ddddc contains 39 entries: Tag Type Name/Value 0x00000001 NEEDED Shared library: [librt.so.1] 0x00000001 NEEDED Shared library: [libcrypt.so.5] 0x00000001 NEEDED Shared library: [libregex.so.1] 0x00000001 NEEDED Shared library: [libcrypto.so.11] 0x00000001 NEEDED Shared library: [libssl.so.11] 0x00000001 NEEDED Shared library: [libm.so.5] 0x00000001 NEEDED Shared library: [libpcreposix.so.0] 0x00000001 NEEDED Shared library: [libpcre.so.1] 0x00000001 NEEDED Shared library: [libkrb5.so.26] 0x00000001 NEEDED Shared library: [libgssapi.so.3] 0x00000001 NEEDED Shared library: [libc++.so.1] 0x00000001 NEEDED Shared library: [libcxxrt.so.1] 0x00000001 NEEDED Shared library: [libgcc_s.so.1] 0x00000001 NEEDED Shared library: [libthr.so.3] 0x00000001 NEEDED Shared library: [libc.so.7] 0x0000001d RUNPATH Library runpath: [/usr/local/lib/heimdal:/usr/lib:/usr/local/lib] 0x0000000c INIT 0xfe0d8 0x0000000d FINI 0x45e0b0 0x00000019 INIT_ARRAY 0x0000001b INIT_ARRAYSZ 292 (bytes) 0x00000004 HASH 0x19c 0x00000005 STRTAB 0x4a520 0x00000006 SYMTAB 0x155c0 0x0000000a STRSZ 580827 (bytes) 0x0000000b SYMENT 16 (bytes) 0x00000015 DEBUG 0x0 0x00000003 PLTGOT 0x5e0c94 0x00000002 PLTRELSZ 4432 (bytes) 0x00000014 PLTREL REL 0x00000017 JMPREL 0xfcf88 0x00000011 REL 0xded48 0x00000012 RELSZ 123456 (bytes) 0x00000013 RELENT 8 (bytes) 0x6ffffffb FLAGS_1 unknown (0x8000000) 0x6ffffffe VERNEED 0xdebe8 0x6fffffff VERNEEDNUM 9 0x6ffffff0 VERSYM 0xd81fc 0x6ffffffa RELCOUNT 13575 0x00000000 NULL 0x0 Finding the "Cannont load PIE" string in "/usr/src/libexec/rtld-elf/rtld.c" So commented out the code: if (obj->z_pie) { _rtld_error("Cannot load PIE binary %s as DSO", obj->path); goto errp; just to get going... # cd /usr/src/libexec/rtld-elf # make -DUSE_K8 clean && make -DUSE_K8 && make -DUSE_K8 install ... cc -O2 -pipe -g0 -ggdb0 -DSTRIP_FBSDID -UDEBUGGING -UDEBUG -DUSB_HAVE_DISABLE_ENUM -O2 -fno-math-errno -fomit-frame-pointer -Wno-error=unused-command-line-argument -Wl,--hash-style=sysv -fno-common -march=haswell -Wall -DFREEBSD_ELF -DIN_RTLD -ffreestanding -I/smallblocks/src/lib/csu/common -I/smallblocks/src/libexec/rtld-elf/amd64 -I/smallblocks/src/libexec/rtld-elf -fpic -DPIC -fvisibility=hidden -mno-mmx -mno-sse -mno-avx -mno-avx2 -msoft-float -DNDEBUG -std=gnu99 -Wsystem-headers -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -Wformat=2 -Wno-format-extra-args -Wmissing-variable-declarations -Wthread-safety -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Qunused-arguments -nostdlib -e .rtld_start -shared -Wl,-Bsymbolic -Wl,-z,defs -Wl,--version-script=Version.map -Wl,-znow -o ld-elf.so.1 rtld_start.o reloc.o rtld.o rtld_lock.o rtld_malloc.o rtld_printf.o map_object.o xmalloc.o debug.o libmap.o -L/usr/obj/smallblocks/src/amd64.amd64/lib/libc -lc_nossp_pic ld: error: unable to find library -lc_nossp_pic cc: error: linker command failed with exit code 1 (use -v to see invocation) *** Error code 1 (Most likely need to build libc - lets skip ahead) Resort to a previous build (tar file) and pull ONLY /libexec/ld-elf.so. Installing this, into the i386 jails -r-xr-xr-x 1 root wheel 134040 20 May 11:39 libexec/ld-elf.so.1 enabled ldd to provide details, the necessary libraries are reachable. # /usr/local/etc/rc.d/squid start Starting squid. Segmentation fault --- squid also fails in an amd64 jail - segmentation fault. Problem 2 - gcc 9.3.0 problem cc1 internal compiler error --------- OK lets try rebuilding the simplest port audio/beep, in an amd64 jail --- beep.o --- cc1: internal compiler error: Segmentation fault libbacktrace could not find executable to open Please submit a full bug report, with preprocessed source if appropriate. See for instructions. *** [beep.o] Error code 1 make[1]: stopped in /var/ports/usr/ports/audio/beep/work/beep 1 error As I force a lot of ports to use gcc9, this was a surprise, especially as gcc9 is one of the few ports that doesn't use esoteric flags like pie, bind or noexecstack. Lets use clang 10 instead. clang 10 builds beep and it runs from /usr/ports/audio/beep/work/stage/usr/local/bin/beep PS I use gcc9 due to the availability of -fstack-clash-protection (which is also in clang 10.1; for another day) Problem 3 - pkg neither installs nor packages --------- ====> Compressing man pages (compress-man) ===> Building package for beep-1.0_1 pkg-static: failed to get the note section pkg-static: failed to get the note section pkg-static: Unable to determine ABI pkg-static: Cannot parse configuration file! *** Error code 1 Stop. This is going to be a problem, as "make -C www/squid clean package" neither builds a package nor performs an installation. Need some mechanism to build/install without the pkg infrastructure... # pkg-static help pkg-static: failed to get the note section pkg-static: failed to get the note section pkg-static: Unable to determine ABI pkg-static: Cannot parse configuration file! Rebuilding pkg with clang 10 and copying /usr/ports/ports-mgmt/pkg/work/stage/usr/local/sbin/pkg-static to /usr/local/sbin/pkg-static, made no difference. Back to squid ------------- OK - lets try a bare bones squid, strip out everything to get a mini-squid (Modify Makefile - Remove all defaults from squid, except GSSAPI_NONE; remove perl5 from USES) Because we can't build a package or install, lets try: # /usr/ports/www/squid/work/stage/usr/local/sbin/squid -h YES!!! (remember though, this is the old /libexec/ld-elf.so.1) The runtime dependencies for "reduced" squid match "fat" squid! Comment out the refresh_patterns in squid.conf (probably why perl is needed) And... squid runs! (Something from Galaxy Quest comes to mind) Reinstate the original (ie recently updated) ld-elf.so.1. Yep squid starts and functions properly, though its interesting that # ldd /usr/local/sbin/squid /usr/local/sbin/squid: ldd: /usr/local/sbin/squid: Cannot load PIE binary /usr/local/sbin/squid as DSO /usr/local/sbin/squid: exit status 1 --- Conclusion ---------- Some folks, perhaps only those that use pie : - an upgrade to 12.1S may cause applications to segfault on both i386 and amd64, these applications will require a rebuild - pkg is problematic, and what in the OS upgrade is causing these messages (& failure) that weren't there yesterday? - gcc9 isn't going to be an easy rebuild as it depends on gmake et al, which won't install... - iterating through: relro, now, pie, noexecstack indicated no contribution to the problem. Seems to be something in image activation... ld-elf ? Next steps: investigate failed to get the note - /usr/ports/ports-mgmt/pkg/work/pkg-1.13.2/libpkg/pkg_elf.c Unable to determine ABI - /var/ports/usr/ports/ports-mgmt/pkg/work/pkg-1.13.2/libpkg/pkg_config.c Cannot parse conf - /var/ports/usr/ports/ports-mgmt/pkg/work/pkg-1.13.2/src/main.c PS Re-enabling kern.elf32 and kern.elf64 and restarting the production applications are functioning happily. Most applications are ok, squid and gcc9 stand out.