From owner-freebsd-hackers@freebsd.org Mon Aug 22 16:14:44 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 777DDBC24F7 for ; Mon, 22 Aug 2016 16:14:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0767F1F43; Mon, 22 Aug 2016 16:14:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u7MGEc0b020507 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 22 Aug 2016 19:14:38 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u7MGEc0b020507 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u7MGEcmX020506; Mon, 22 Aug 2016 19:14:38 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 22 Aug 2016 19:14:37 +0300 From: Konstantin Belousov To: Matthew Seaman Cc: freebsd-hackers@freebsd.org Subject: Re: How do I debug this? Message-ID: <20160822161437.GC83214@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-1.5 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED,URI_NOVOWEL autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Aug 2016 16:14:44 -0000 On Mon, Aug 22, 2016 at 04:54:22PM +0100, Matthew Seaman wrote: > > We've got an intermittent crash bug with postgresql92-server-9.2.17_2 > which seems to be occurring in process startup: > > # gdb -c /usr/local/pgsql/data/postgres.core /usr/local/bin/postgres > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > Core was generated by `postgres'. > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libthr.so.3...done. > Loaded symbols for /lib/libthr.so.3 > Reading symbols from /usr/local/lib/libxml2.so.2...done. > Loaded symbols for /usr/local/lib/libxml2.so.2 > Reading symbols from /usr/local/lib/libicuuc.so.55...done. > Loaded symbols for /usr/local/lib/libicuuc.so.55 > Reading symbols from /usr/local/lib/libicui18n.so.55...done. > Loaded symbols for /usr/local/lib/libicui18n.so.55 > Reading symbols from /usr/local/lib/libssl.so.8...done. > Loaded symbols for /usr/local/lib/libssl.so.8 > Reading symbols from /usr/local/lib/libcrypto.so.8...done. > Loaded symbols for /usr/local/lib/libcrypto.so.8 > Reading symbols from /lib/libm.so.5...done. > Loaded symbols for /lib/libm.so.5 > Reading symbols from /lib/libc.so.7...done. > Loaded symbols for /lib/libc.so.7 > Reading symbols from /lib/libz.so.6...done. > Loaded symbols for /lib/libz.so.6 > Reading symbols from /usr/lib/liblzma.so.5...done. > Loaded symbols for /usr/lib/liblzma.so.5 > Reading symbols from /usr/local/lib/libicudata.so.55...done. > Loaded symbols for /usr/local/lib/libicudata.so.55 > Reading symbols from /usr/lib/libc++.so.1...done. > Loaded symbols for /usr/lib/libc++.so.1 > Reading symbols from /lib/libcxxrt.so.1...done. > Loaded symbols for /lib/libcxxrt.so.1 > Reading symbols from /lib/libgcc_s.so.1...done. > Loaded symbols for /lib/libgcc_s.so.1 > Reading symbols from /usr/local/lib/postgresql/auto_explain.so...done. > Loaded symbols for /usr/local/lib/postgresql/auto_explain.so > Reading symbols from /usr/local/lib/postgresql/plpgsql.so...done. > Loaded symbols for /usr/local/lib/postgresql/plpgsql.so > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x0000000800d1cd0a in _r_debug_postinit () from /libexec/ld-elf.so.1 > [New Thread 805806400 (LWP 100362/)] > (gdb) bt full > #0 0x0000000800d1cd0a in _r_debug_postinit () from /libexec/ld-elf.so.1 > No symbol table info available. > #1 0x0000000800d1a46d in .text () from /libexec/ld-elf.so.1 > No symbol table info available. > #2 0x0000000804ffe174 in ?? () from /lib/libgcc_s.so.1 > No symbol table info available. > #3 0x00007fffffffe410 in ?? () > No symbol table info available. > #4 0x00000008050078f1 in ?? () from /lib/libgcc_s.so.1 > No symbol table info available. > #5 0x00007fffffffe410 in ?? () > No symbol table info available. > #6 0x0000000800d1d9a8 in dlclose () from /libexec/ld-elf.so.1 > No symbol table info available. > Previous frame inner to this frame (corrupt stack?) > > Application is: > > postgresql92-server-9.2.17_2 > > On: > > % freebsd-version -ku > 10.3-RELEASE-p4 > 10.3-RELEASE-p5 > > This is apparently a postgres child process crashing on startup > somewhere before where main() gets called. Doesn't happen often -- only > when the server is reasonably busy and then it will take somewhere > between a day and a week between incidents. We can't correlate it with > any particular SQL being run. This was a production server so we've had > to move the customers assigned to it back to some slightly older but > very hardware but running the same version of postgresql (same package > compiled on our own pkgrepo) and same 10.3-RELEASE-p4/p5 -- where they > are running quite happily. > Compile rtld/libc/libthr and postgres itself with debugging symbols and re-obtain the backtrace, for start.