From owner-freebsd-current@freebsd.org Sun Mar 13 18:12:43 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93874ACF78C for ; Sun, 13 Mar 2016 18:12:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 25517CAC for ; Sun, 13 Mar 2016 18:12:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u2DICZgh003503 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 13 Mar 2016 20:12:35 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u2DICZgh003503 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u2DICYig003502; Sun, 13 Mar 2016 20:12:34 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 13 Mar 2016 20:12:34 +0200 From: Konstantin Belousov To: Larry Rosenman Cc: Freebsd current Subject: Re: Crashes in libthr? Message-ID: <20160313181234.GI1741@kib.kiev.ua> References: <70c63304fceddc1a91e16d63152ff33a@thebighonker.lerctr.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <70c63304fceddc1a91e16d63152ff33a@thebighonker.lerctr.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Mar 2016 18:12:43 -0000 On Sun, Mar 13, 2016 at 11:16:20AM -0500, Larry Rosenman wrote: > I updated one of my servers, and WHILE DOING THE INSTALLWORLD, I get > segfaults. > > ANY multithreaded program crashes. > > I reverted libthr, and it's fine. > > borg.lerctr.org / # gdb -c zfs.core /sbin/zfs > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you > are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "amd64-marcel-freebsd"... > Core was generated by `zfs'. > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libjail.so.1...Reading symbols from > /usr/lib/debug//lib/libjail.so.1.debug...done. > done. > Loaded symbols for /lib/libjail.so.1 > Reading symbols from /lib/libnvpair.so.2...Reading symbols from > /usr/lib/debug//lib/libnvpair.so.2.debug...done. > done. > Loaded symbols for /lib/libnvpair.so.2 > Reading symbols from /lib/libuutil.so.2...Reading symbols from > /usr/lib/debug//lib/libuutil.so.2.debug...done. > done. > Loaded symbols for /lib/libuutil.so.2 > Reading symbols from /lib/libzfs_core.so.2...Reading symbols from > /usr/lib/debug//lib/libzfs_core.so.2.debug...done. > done. > Loaded symbols for /lib/libzfs_core.so.2 > Reading symbols from /lib/libzfs.so.2...Reading symbols from > /usr/lib/debug//lib/libzfs.so.2.debug...done. > done. > Loaded symbols for /lib/libzfs.so.2 > Reading symbols from /lib/libc.so.7...Reading symbols from > /usr/lib/debug//lib/libc.so.7.debug...done. > done. > Loaded symbols for /lib/libc.so.7 > Reading symbols from /lib/libmd.so.6...Reading symbols from > /usr/lib/debug//lib/libmd.so.6.debug...done. > done. > Loaded symbols for /lib/libmd.so.6 > Reading symbols from /lib/libumem.so.2...Reading symbols from > /usr/lib/debug//lib/libumem.so.2.debug...done. > done. > Loaded symbols for /lib/libumem.so.2 > Reading symbols from /lib/libutil.so.9...Reading symbols from > /usr/lib/debug//lib/libutil.so.9.debug...done. > done. > Loaded symbols for /lib/libutil.so.9 > Reading symbols from /lib/libm.so.5...Reading symbols from > /usr/lib/debug//lib/libm.so.5.debug...done. > done. > Loaded symbols for /lib/libm.so.5 > Reading symbols from /lib/libavl.so.2...Reading symbols from > /usr/lib/debug//lib/libavl.so.2.debug...done. > done. > Loaded symbols for /lib/libavl.so.2 > Reading symbols from /lib/libbsdxml.so.4...Reading symbols from > /usr/lib/debug//lib/libbsdxml.so.4.debug...done. > done. > Loaded symbols for /lib/libbsdxml.so.4 > Reading symbols from /lib/libgeom.so.5...Reading symbols from > /usr/lib/debug//lib/libgeom.so.5.debug...done. > done. > Loaded symbols for /lib/libgeom.so.5 > Reading symbols from /lib/libz.so.6...Reading symbols from > /usr/lib/debug//lib/libz.so.6.debug...done. > done. > Loaded symbols for /lib/libz.so.6 > Reading symbols from /lib/libthr.so.3...done. > Loaded symbols for /lib/libthr.so.3 Why all libs have debug symbols, while your most interesting one, libthr.so.3, does not ? > Reading symbols from /lib/libsbuf.so.6...Reading symbols from > /usr/lib/debug//lib/libsbuf.so.6.debug...done. > done. > Loaded symbols for /lib/libsbuf.so.6 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x0000000802703f81 in __pthread_cxa_finalize () from > /lib/libthr.so.3 > [New LWP 100957] > (gdb) bt > #0 0x0000000802703f81 in __pthread_cxa_finalize () from > /lib/libthr.so.3 > #1 0x0000000802703e85 in __pthread_cxa_finalize () from > /lib/libthr.so.3 > #2 0x0000000802707052 in ?? () from /lib/libthr.so.3 > #3 0x000000080063fc00 in ?? () > #4 0x00007fffffffe638 in ?? () > #5 0x00007fffffffe5b0 in ?? () > #6 0x00000008026f8fd6 in atoi@plt () from /lib/libthr.so.3 > #7 0x00007fffffffe5b0 in ?? () > #8 0x000000080061adfd in r_debug_state () from /libexec/ld-elf.so.1 > Previous frame inner to this frame (corrupt stack?) > (gdb) > > old SVN: r296103 > new SVN: r296796M > (The M is a nd6 patch from markj@) > > this was a FULL buildworld/buildkernel. If you cd lib/libthr and do make clean all install DEBUG_FLAGS=-g on the broken world, does it fix the problem ? If not, do debugging symbols from libthr appear accessible to gdb at least ? Try this to get useful backtrace with source lines information.