From owner-freebsd-current@freebsd.org Sun Mar 13 19:10:58 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F025BACE04F for ; Sun, 13 Mar 2016 19:10:58 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CFF1FF3A for ; Sun, 13 Mar 2016 19:10:58 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date: Content-Transfer-Encoding:Content-Type:MIME-Version; bh=UtfVr2qXI2GP6ecoYM7ifJDMAhpboIo4kgHOjQi+l4c=; b=iwsoRerVHcgs8334yM3ZMX21oc eyarkkiXh2QQJGB5nP1F+m3cIAuDwvKmuS3D/2h8TW3dQwMtsopmI3S1EndSQ3T4CpG1R7590mQGi Lq4T8B+1dQi2JXKSBTmbuqoEftUMJ7Da3fGDyREriaG1n74PZzNsGOrZR/3cmJNUjUO0=; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]:34299 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.86_2 (FreeBSD)) (envelope-from ) id 1afBPm-000DLO-2q; Sun, 13 Mar 2016 14:10:58 -0500 Received: from 2605:6000:ec17:203:96d:66a8:97b8:1157 by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Sun, 13 Mar 2016 14:10:58 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sun, 13 Mar 2016 14:10:58 -0500 From: Larry Rosenman To: Konstantin Belousov Cc: Freebsd current Subject: Re: Crashes in libthr? In-Reply-To: <20160313185852.GJ1741@kib.kiev.ua> References: <70c63304fceddc1a91e16d63152ff33a@thebighonker.lerctr.org> <20160313181234.GI1741@kib.kiev.ua> <276f6b11da9a005256d24fa5b37d7d5f@thebighonker.lerctr.org> <20160313185852.GJ1741@kib.kiev.ua> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.1.4 X-Spam-Score: -1.0 (-) X-LERCTR-Spam-Score: -1.0 (-) X-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 X-LERCTR-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1, SHORTCIRCUIT=-0.0001 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Mar 2016 19:10:59 -0000 On 2016-03-13 13:58, Konstantin Belousov wrote: > On Sun, Mar 13, 2016 at 01:32:20PM -0500, Larry Rosenman wrote: >> On 2016-03-13 13:12, Konstantin Belousov wrote: >> > On Sun, Mar 13, 2016 at 11:16:20AM -0500, Larry Rosenman wrote: >> >> I updated one of my servers, and WHILE DOING THE INSTALLWORLD, I get >> >> segfaults. >> >> >> >> ANY multithreaded program crashes. >> >> >> >> I reverted libthr, and it's fine. >> >> >> >> borg.lerctr.org / # gdb -c zfs.core /sbin/zfs >> >> GNU gdb 6.1.1 [FreeBSD] >> >> Copyright 2004 Free Software Foundation, Inc. >> >> GDB is free software, covered by the GNU General Public License, and >> >> you >> >> are >> >> welcome to change it and/or distribute copies of it under certain >> >> conditions. >> >> Type "show copying" to see the conditions. >> >> There is absolutely no warranty for GDB. Type "show warranty" for >> >> details. >> >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Core was generated by `zfs'. >> >> Program terminated with signal 11, Segmentation fault. >> >> Reading symbols from /lib/libjail.so.1...Reading symbols from >> >> /usr/lib/debug//lib/libjail.so.1.debug...done. >> >> done. >> >> Loaded symbols for /lib/libjail.so.1 >> >> Reading symbols from /lib/libnvpair.so.2...Reading symbols from >> >> /usr/lib/debug//lib/libnvpair.so.2.debug...done. >> >> done. >> >> Loaded symbols for /lib/libnvpair.so.2 >> >> Reading symbols from /lib/libuutil.so.2...Reading symbols from >> >> /usr/lib/debug//lib/libuutil.so.2.debug...done. >> >> done. >> >> Loaded symbols for /lib/libuutil.so.2 >> >> Reading symbols from /lib/libzfs_core.so.2...Reading symbols from >> >> /usr/lib/debug//lib/libzfs_core.so.2.debug...done. >> >> done. >> >> Loaded symbols for /lib/libzfs_core.so.2 >> >> Reading symbols from /lib/libzfs.so.2...Reading symbols from >> >> /usr/lib/debug//lib/libzfs.so.2.debug...done. >> >> done. >> >> Loaded symbols for /lib/libzfs.so.2 >> >> Reading symbols from /lib/libc.so.7...Reading symbols from >> >> /usr/lib/debug//lib/libc.so.7.debug...done. >> >> done. >> >> Loaded symbols for /lib/libc.so.7 >> >> Reading symbols from /lib/libmd.so.6...Reading symbols from >> >> /usr/lib/debug//lib/libmd.so.6.debug...done. >> >> done. >> >> Loaded symbols for /lib/libmd.so.6 >> >> Reading symbols from /lib/libumem.so.2...Reading symbols from >> >> /usr/lib/debug//lib/libumem.so.2.debug...done. >> >> done. >> >> Loaded symbols for /lib/libumem.so.2 >> >> Reading symbols from /lib/libutil.so.9...Reading symbols from >> >> /usr/lib/debug//lib/libutil.so.9.debug...done. >> >> done. >> >> Loaded symbols for /lib/libutil.so.9 >> >> Reading symbols from /lib/libm.so.5...Reading symbols from >> >> /usr/lib/debug//lib/libm.so.5.debug...done. >> >> done. >> >> Loaded symbols for /lib/libm.so.5 >> >> Reading symbols from /lib/libavl.so.2...Reading symbols from >> >> /usr/lib/debug//lib/libavl.so.2.debug...done. >> >> done. >> >> Loaded symbols for /lib/libavl.so.2 >> >> Reading symbols from /lib/libbsdxml.so.4...Reading symbols from >> >> /usr/lib/debug//lib/libbsdxml.so.4.debug...done. >> >> done. >> >> Loaded symbols for /lib/libbsdxml.so.4 >> >> Reading symbols from /lib/libgeom.so.5...Reading symbols from >> >> /usr/lib/debug//lib/libgeom.so.5.debug...done. >> >> done. >> >> Loaded symbols for /lib/libgeom.so.5 >> >> Reading symbols from /lib/libz.so.6...Reading symbols from >> >> /usr/lib/debug//lib/libz.so.6.debug...done. >> >> done. >> >> Loaded symbols for /lib/libz.so.6 >> >> Reading symbols from /lib/libthr.so.3...done. >> >> Loaded symbols for /lib/libthr.so.3 >> > Why all libs have debug symbols, while your most interesting one, >> > libthr.so.3, does not ? >> > >> >> Reading symbols from /lib/libsbuf.so.6...Reading symbols from >> >> /usr/lib/debug//lib/libsbuf.so.6.debug...done. >> >> done. >> >> Loaded symbols for /lib/libsbuf.so.6 >> >> Reading symbols from /libexec/ld-elf.so.1...done. >> >> Loaded symbols for /libexec/ld-elf.so.1 >> >> #0 0x0000000802703f81 in __pthread_cxa_finalize () from >> >> /lib/libthr.so.3 >> >> [New LWP 100957] >> >> (gdb) bt >> >> #0 0x0000000802703f81 in __pthread_cxa_finalize () from >> >> /lib/libthr.so.3 >> >> #1 0x0000000802703e85 in __pthread_cxa_finalize () from >> >> /lib/libthr.so.3 >> >> #2 0x0000000802707052 in ?? () from /lib/libthr.so.3 >> >> #3 0x000000080063fc00 in ?? () >> >> #4 0x00007fffffffe638 in ?? () >> >> #5 0x00007fffffffe5b0 in ?? () >> >> #6 0x00000008026f8fd6 in atoi@plt () from /lib/libthr.so.3 >> >> #7 0x00007fffffffe5b0 in ?? () >> >> #8 0x000000080061adfd in r_debug_state () from /libexec/ld-elf.so.1 >> >> Previous frame inner to this frame (corrupt stack?) >> >> (gdb) >> >> >> >> old SVN: r296103 >> >> new SVN: r296796M >> >> (The M is a nd6 patch from markj@) >> >> >> >> this was a FULL buildworld/buildkernel. >> > >> > If you cd lib/libthr and do >> > make clean all install DEBUG_FLAGS=-g >> > on the broken world, does it fix the problem ? If not, do debugging >> > symbols from libthr appear accessible to gdb at least ? Try this to >> > get useful backtrace with source lines information. >> ar crashes linking the library...... > > ar does not link library, it archives .o into .a archive. > That said, neither ar nor ld (I am not sure whether 'ar' or 'linking' > is a thinko in the sentence above) do not depend on libthr. You have > something more fundamental broken. > > I put the libthr.so.3 built with the debugging symbols on amd64, using > head > r296779, at https://people.freebsd.org/~kib/misc/libthr.so.3 . Try > with > it by manually copying the file to /lib. If still crashing, it should > show the line numbers. no dice. (gdb) borg.lerctr.org /home/ler $ sudo gdb -c ar.core /usr/bin/ar Password: GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `ar'. Program terminated with signal 11, Segmentation fault. #0 0x000000000042c271 in _thr_rtld_init () [New Thread 800a1b000 (LWP 100317/)] (gdb) bt #0 0x000000000042c271 in _thr_rtld_init () #1 0x000000000042c19a in _libpthread_init () #2 0x00000000004eaa82 in __do_global_ctors_aux () #3 0x0000000000400196 in _init () #4 0x00007fffffffebd0 in ?? () #5 0x00000000004002a6 in _start () #6 0x0000000000000000 in ?? () (gdb) -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 7011 W Parmer Ln, Apt 1115, Austin, TX 78729-6961