Date: Mon, 22 Aug 2016 19:14:37 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Matthew Seaman <matthew@FreeBSD.org> Cc: freebsd-hackers@freebsd.org Subject: Re: How do I debug this? Message-ID: <20160822161437.GC83214@kib.kiev.ua> In-Reply-To: <a6b6b604-80ad-926a-5d85-871e9dd7ebbe@freebsd.org> References: <a6b6b604-80ad-926a-5d85-871e9dd7ebbe@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Aug 22, 2016 at 04:54:22PM +0100, Matthew Seaman wrote: > > We've got an intermittent crash bug with postgresql92-server-9.2.17_2 > which seems to be occurring in process startup: > > # gdb -c /usr/local/pgsql/data/postgres.core /usr/local/bin/postgres > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > Core was generated by `postgres'. > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libthr.so.3...done. > Loaded symbols for /lib/libthr.so.3 > Reading symbols from /usr/local/lib/libxml2.so.2...done. > Loaded symbols for /usr/local/lib/libxml2.so.2 > Reading symbols from /usr/local/lib/libicuuc.so.55...done. > Loaded symbols for /usr/local/lib/libicuuc.so.55 > Reading symbols from /usr/local/lib/libicui18n.so.55...done. > Loaded symbols for /usr/local/lib/libicui18n.so.55 > Reading symbols from /usr/local/lib/libssl.so.8...done. > Loaded symbols for /usr/local/lib/libssl.so.8 > Reading symbols from /usr/local/lib/libcrypto.so.8...done. > Loaded symbols for /usr/local/lib/libcrypto.so.8 > Reading symbols from /lib/libm.so.5...done. > Loaded symbols for /lib/libm.so.5 > Reading symbols from /lib/libc.so.7...done. > Loaded symbols for /lib/libc.so.7 > Reading symbols from /lib/libz.so.6...done. > Loaded symbols for /lib/libz.so.6 > Reading symbols from /usr/lib/liblzma.so.5...done. > Loaded symbols for /usr/lib/liblzma.so.5 > Reading symbols from /usr/local/lib/libicudata.so.55...done. > Loaded symbols for /usr/local/lib/libicudata.so.55 > Reading symbols from /usr/lib/libc++.so.1...done. > Loaded symbols for /usr/lib/libc++.so.1 > Reading symbols from /lib/libcxxrt.so.1...done. > Loaded symbols for /lib/libcxxrt.so.1 > Reading symbols from /lib/libgcc_s.so.1...done. > Loaded symbols for /lib/libgcc_s.so.1 > Reading symbols from /usr/local/lib/postgresql/auto_explain.so...done. > Loaded symbols for /usr/local/lib/postgresql/auto_explain.so > Reading symbols from /usr/local/lib/postgresql/plpgsql.so...done. > Loaded symbols for /usr/local/lib/postgresql/plpgsql.so > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x0000000800d1cd0a in _r_debug_postinit () from /libexec/ld-elf.so.1 > [New Thread 805806400 (LWP 100362/)] > (gdb) bt full > #0 0x0000000800d1cd0a in _r_debug_postinit () from /libexec/ld-elf.so.1 > No symbol table info available. > #1 0x0000000800d1a46d in .text () from /libexec/ld-elf.so.1 > No symbol table info available. > #2 0x0000000804ffe174 in ?? () from /lib/libgcc_s.so.1 > No symbol table info available. > #3 0x00007fffffffe410 in ?? () > No symbol table info available. > #4 0x00000008050078f1 in ?? () from /lib/libgcc_s.so.1 > No symbol table info available. > #5 0x00007fffffffe410 in ?? () > No symbol table info available. > #6 0x0000000800d1d9a8 in dlclose () from /libexec/ld-elf.so.1 > No symbol table info available. > Previous frame inner to this frame (corrupt stack?) > > Application is: > > postgresql92-server-9.2.17_2 > > On: > > % freebsd-version -ku > 10.3-RELEASE-p4 > 10.3-RELEASE-p5 > > This is apparently a postgres child process crashing on startup > somewhere before where main() gets called. Doesn't happen often -- only > when the server is reasonably busy and then it will take somewhere > between a day and a week between incidents. We can't correlate it with > any particular SQL being run. This was a production server so we've had > to move the customers assigned to it back to some slightly older but > very hardware but running the same version of postgresql (same package > compiled on our own pkgrepo) and same 10.3-RELEASE-p4/p5 -- where they > are running quite happily. > Compile rtld/libc/libthr and postgres itself with debugging symbols and re-obtain the backtrace, for start.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160822161437.GC83214>