From owner-freebsd-stable@freebsd.org Sat Jun 13 16:14:53 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 307ED33C822 for ; Sat, 13 Jun 2020 16:14:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49kjNX12Vkz4Dj7; Sat, 13 Jun 2020 16:14:51 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id 05DGEfRo091074 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 13 Jun 2020 19:14:44 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 05DGEfRo091074 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id 05DGEcCj091072; Sat, 13 Jun 2020 19:14:38 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 13 Jun 2020 19:14:38 +0300 From: Konstantin Belousov To: Dewayne Geraghty Cc: freebsd-stable@freebsd.org, emaste@freebsd.org Subject: Re: Upgrading to 12.1S 362003 - a few issues Message-ID: <20200613161438.GT48478@kib.kiev.ua> References: <978b8cb5-de88-4264-25ec-e7ecaaf80c75@heuristicsystems.com.au> <20200613013450.GS48478@kib.kiev.ua> <355bf1bc-82cf-a558-16df-1c7c2503c7a8@heuristicsystems.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <355bf1bc-82cf-a558-16df-1c7c2503c7a8@heuristicsystems.com.au> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 49kjNX12Vkz4Dj7 X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [2.43 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; NEURAL_SPAM_MEDIUM(0.77)[0.772]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.93)[0.933]; NEURAL_SPAM_LONG(0.73)[0.728]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jun 2020 16:14:53 -0000 On Sat, Jun 13, 2020 at 12:34:46PM +1000, Dewayne Geraghty wrote: > Hi Konstantin, > I did try ktrace/kdump but kdump complained of "data too short". Using > your suggestion about LD_DEBUG nearly caused me to fall off my chair. I > think this is most relevant: > > # setenv LD_DEBUG 1 > # ktrace -f /tmp/sq3.kt /usr/local/sbin/squid start > /libexec/ld-elf.so.1 is initialized, base address = 0x8aded000 > RTLD dynamic = 0x8ae0f6c0 > RTLD pltgot = 0 > initializing thread locks > _rtld_thread_init: done > processing main program's program header > note osrel 1201517 > note fctl0 0 > note crt_no_init > AT_EXECPATH 0xffffdfe0 /usr/bin/ktrace > obj_main path /usr/bin/ktrace > Filling in DT_DEBUG entry > Ignoring d_tag 1879048186 = 0x6ffffffa > /usr/bin/ktrace valid_hash_sysv 1 valid_hash_gnu 0 dynsymcount 30 > lm_init("(null)") > loading LD_PRELOAD libraries > loading needed objects > Searching for "libc.so.7" > search_library_pathfds('libc.so.7', '(null)', fdp) > lm_find("/usr/bin/ktrace", "/lib") > lmp_find("/usr/bin/ktrace") > lmp_find("$DEFAULT$") > Trying "/lib/libc.so.7" > Opened "/lib/libc.so.7", fd 3 > loading "/lib/libc.so.7" > Ignoring d_tag 1879048186 = 0x6ffffffa > /lib/libc.so.7 valid_hash_sysv 1 valid_hash_gnu 1 dynsymcount 3126 > 0x6d20f000 .. 0x6d3eafff: /lib/libc.so.7 > checking for required versions > initializing initial thread local storage offsets > relocating "/usr/bin/ktrace" > reloc_jmpslot: *0x4b177c04 = 0x6d39bac0 > reloc_jmpslot: *0x4b177c08 = 0x6d3b6480 > ... [ Lots of these ] > reloc_jmpslot: *0x9d4859fc = 0x9d46e560 > reloc_jmpslot: *0x9d485a00 = 0x9d47a428 > reloc_jmpslot: *0x9d485a04 = 0x9d47a448 > relocating "/usr/local/lib/heimdal/libasn1.so.8" > relocating "/usr/local/lib/heimdal/libwind.so.0" > relocating "/usr/local/lib/heimdal/libheimbase.so.1" > relocating "/usr/local/lib/heimdal/libhx509.so.5" > relocating "/usr/local/lib/heimdal/libhcrypto.so.4" > relocating "/usr/local/lib/heimdal/libheimsqlite.so.0" > relocating "/usr/local/lib/heimdal/libcom_err.so.1" > relocating "/usr/local/lib/heimdal/libroken.so.18" > relocating "/usr/local/lib/libintl.so.8" > relocating "/usr/local/lib/heimdal/libheimntlm.so.0" > doing copy relocations > initializing initial thread local storage > initializing key program variables > "__progname": *0x73be38b8 <-- 0xffffdd44 > "environ": *0x73cbbc08 <-- 0xfffc69cc > "__elf_aux_vector": *0x9d4a6a5c <-- 0xfffc6a4c > resolving ifuncs > reloc_jmpslot: *0x9d484d88 = 0x9d352cc0 > reloc_jmpslot: *0x9d484d8c = 0x9d352c70 > reloc_jmpslot: *0x9cb16e04 = 0x9d352c70 > reloc_jmpslot: *0x9cb16e0c = 0x9d352cc0 > calling init function for /lib/libc.so.7 at 0x9d47c440 > calling init function for /lib/libc.so.7 at 0x9d40fc20 > calling init function for /lib/libc.so.7 at 0x9d3f0f40 > calling init function for /lib/libthr.so.3 at 0x9cb15210 > calling init function for /lib/libthr.so.3 at 0x9cb10cb0 > _rtld_thread_init: done > calling init function for /lib/libgcc_s.so.1 at 0x9bfad540 > calling init function for /lib/libgcc_s.so.1 at 0x9bfa3de0 > calling init function for /lib/libcxxrt.so.1 at 0x9b698480 > calling init function for /usr/lib/libc++.so.1 at 0x9ab43d90 > calling init function for /usr/lib/libc++.so.1 at 0x9aae3e40 > calling init function for /lib/libcrypt.so.5 at 0x965d3210 > calling init function for /usr/local/lib/libcrypto.so.11 at 0x96e9f420 > calling init function for /usr/local/lib/heimdal/libroken.so.18 at > 0xa32b6f60 > calling init function for /usr/local/lib/libintl.so.8 at 0xa23cb104 > calling init function for /usr/local/lib/heimdal/libheimbase.so.1 at > 0x9ed15de0 > calling init function for /usr/local/lib/heimdal/libcom_err.so.1 at > 0x9fec3768 > calling init function for /usr/local/lib/heimdal/libasn1.so.8 at 0x9e23f054 > calling init function for /usr/local/lib/heimdal/libhcrypto.so.4 at > 0x9fb5641c > calling init function for /usr/local/lib/heimdal/libheimsqlite.so.0 at > 0xa1b9603c > calling init function for /usr/local/lib/heimdal/libwind.so.0 at 0x9f77a990 > calling init function for /usr/local/lib/heimdal/libhx509.so.5 at 0xa0c4e3b4 > calling init function for /usr/local/lib/heimdal/libkrb5.so.26 at 0x99527188 > calling init function for /usr/local/lib/heimdal/libheimntlm.so.0 at > 0xa498c814 > calling init function for /usr/local/lib/heimdal/libgssapi.so.3 at > 0x9a45bfd0 > calling init function for /usr/local/lib/libpcre.so.1 at 0x97550ca8 > calling init function for /usr/local/lib/libpcreposix.so.0 at 0x9728e518 > calling init function for /lib/libm.so.5 at 0x97139f10 > calling init function for /usr/local/lib/libssl.so.11 at 0x98579078 > calling init function for /usr/lib/libregex.so.1 at 0x95e8f0a0 > calling init function for /usr/lib/librt.so.1 at 0x953ca240 > loading filtees > enforcing main obj relro > transferring control to program entry point = 0x73757360 So the app code started executing. > "atexit" in "squid" ==> 0x9d457ac0 in "libc.so.7" > reloc_jmpslot: *0x73be32b4 = 0x9d457ac0 > "_ZSt13set_terminatePFvvE" in "squid" ==> 0x9b697a30 in "libcxxrt.so.1" > reloc_jmpslot: *0x73be3794 = 0x9b697a30 > "strlen" in "squid" ==> 0x9d4750b0 in "libc.so.7" > reloc_jmpslot: *0x73be3108 = 0x9d4750b0 > "malloc" in "squid" ==> 0x9d3e6620 in "libc.so.7" > reloc_jmpslot: *0x73be3820 = 0x9d3e6620 > "memcpy" in "squid" ==> 0x9d47aa20 in "libc.so.7" > reloc_jmpslot: *0x73be3500 = 0x9d47aa20 > "_Znwj" in "squid" ==> 0x9ab437c0 in "libc++.so.1" > reloc_jmpslot: *0x73be3578 = 0x9ab437c0 > "free" in "squid" ==> 0x9d3e9ea0 in "libc.so.7" > reloc_jmpslot: *0x73be3790 = 0x9d3e9ea0 > "_ZdlPv" in "squid" ==> 0x9ab438d0 in "libc++.so.1" > reloc_jmpslot: *0x73be32b8 = 0x9ab438d0 > "getopt_long" in "squid" ==> 0x9d3515f0 in "libc.so.7" > reloc_jmpslot: *0x73be3614 = 0x9d3515f0 > Segmentation fault > > I've placed the ktrace (sq3.kt.xz) and LD_DEBUG output (sq.deb.xz) at > http://www.heuristicsystems.com/squid4K No need, the problem occurs somewhere in user code. Try to build the binary with debug symbols and look into the coredump with gdb. > > Pity about ldd, but I appreciate the complexity & the effort. I'm also > comforted that it isn't an error, rather the evolution of ldd :) ldd should be fixed after https://reviews.freebsd.org/D25257 is committed.