From owner-freebsd-bugs@FreeBSD.ORG Thu Jan 29 14:44:22 2015 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E4D2C828 for ; Thu, 29 Jan 2015 14:44:22 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CBB9F343 for ; Thu, 29 Jan 2015 14:44:22 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t0TEiMk8043464 for ; Thu, 29 Jan 2015 14:44:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 197174] panic: vm_radix_insert: key 473 is already present Date: Thu, 29 Jan 2015 14:44:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: vivek@khera.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2015 14:44:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=197174 Bug ID: 197174 Summary: panic: vm_radix_insert: key 473 is already present Product: Base System Version: 10.0-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: vivek@khera.org Created attachment 152336 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=152336&action=edit dmesg.boot from d06 server Recently two identical boxes I have started issuing the panic: panic: vm_radix_insert: key 473 is already present after which the entire system is locked up. The serial console is unresponsive and my only option is to power cycle the system. The serial console emits the following at the time of panic. All four panics (two on each box so far) have been identical other than the key and CPU number identified in the initial panic line. The dmesg from one of the boxes is attached. The first such panic occurred after 300 days of uptime. The sole purpose of this machine is running Postgres 9.2 server. The other things it does is run NRPE for nagios monitoring and slony1 for database replication. The database lives on a ZFS mirror file system. I'm pretty sure it has something to do with the load on the server since only the active system will panic; the twin box is just a backup. I swapped roles after the second panic on the primary system to see if it was hardware related. I clearly seems software related as the other system started to panic once it was the master. I posted about this a week ago but got no response. Since then I have had one more panic. https://lists.freebsd.org/pipermail/freebsd-questions/2015-January/263732.html I upgraded my original machine to FreeBSD 10.1 and Postgres 9.4, and will keep an eye out for panics with that configuration as well. Here are the panic lines recorded on the serial console: Jan 15 00:21:51 ts-prv src_dev_log@ts Buffering: S9.d05 [panic: vm_radix_insert: key 46f is already present cpuid = 9 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe3fcf1b1820 ] Jan 15 00:21:51 ts-prv src_dev_log@ts Buffering: S9.d05 [kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf1b18d0 panic() at panic+0x155/frame 0xfffffe3fcf1b1950 ] Jan 15 00:21:51 ts-prv src_dev_log@ts Buffering: S9.d05 [vm_radix_insert() at vm_radix_insert+0x2ed/frame 0xfffffe3fcf1b19b0 vm_page_cache() at vm_page_cache+0x121/frame 0xfffffe3fcf1b19f0 ] Jan 15 00:21:51 ts-prv src_dev_log@ts Buffering: S9.d05 [vm_pageout() at vm_pageout+0x8f7/frame 0xfffffe3fcf1b1a70 fork_exit() at fork_exit+0x9a/frame 0xfffffe3fcf1b1ab0 ] Jan 15 00:21:52 ts-prv src_dev_log@ts Buffering: S9.d05 [fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe3fcf1b1ab0 --- trap 0, rip = 0, rsp = 0xfffffe3fcf1b1b70, rbp = 0 --- ] Jan 17 01:13:44 ts-prv src_dev_log@ts Buffering: S9.d05 [panic: vm_radix_insert: key 46f is already present cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe3fcf1b1820 ] Jan 17 01:13:44 ts-prv src_dev_log@ts Buffering: S9.d05 [kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf1b18d0 panic() at panic+0x155/frame 0xfffffe3fcf1b1950 ] Jan 17 01:13:44 ts-prv src_dev_log@ts Buffering: S9.d05 [vm_radix_insert() at vm_radix_insert+0x2ed/frame 0xfffffe3fcf1b19b0 vm_page_cache() at vm_page_cache+0x121/frame 0xfffffe3fcf1b19f0 ] Jan 17 01:13:44 ts-prv src_dev_log@ts Buffering: S9.d05 [vm_pageout() at vm_pageout+0x8f7/frame 0xfffffe3fcf1b1a70 fork_exit() at fork_exit+0x9a/frame 0xfffffe3fcf1b1ab0 ] Jan 17 01:13:44 ts-prv src_dev_log@ts Buffering: S9.d05 [fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe3fcf1b1ab0 --- trap 0, rip = 0, rsp = 0xfffffe3fcf1b1b70, rbp = 0 --- ] from the second machine: Jan 21 23:15:51 ts-prv src_dev_log@ts Buffering: S13.d06 [panic: vm_radix_insert: key 473 is already present cpuid = 11 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/fra] Jan 21 23:15:51 ts-prv src_dev_log@ts Buffering: S13.d06 [me 0xfffffe3fcf1b2820 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf1b28d0 panic() at panic+0x155/frame 0xfffffe3fcf1b2950 ] Jan 21 23:15:51 ts-prv src_dev_log@ts Buffering: S13.d06 [vm_radix_insert() at vm_radix_insert+0x2ed/frame 0xfffffe3fcf1b29b0 vm_page_cache() at vm_page_cache+0x121/frame 0xfffffe3fcf1b29f0 ] Jan 21 23:15:51 ts-prv src_dev_log@ts Buffering: S13.d06 [vm_pageout() at vm_pageout+0x8f7/frame 0xfffffe3fcf1b2a70 fork_exit() at fork_exit+0x9a/frame 0xfffffe3fcf1b2ab0 ] Jan 21 23:15:51 ts-prv src_dev_log@ts Buffering: S13.d06 [fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe3fcf1b2ab0 --- trap 0, rip = 0, rsp = 0xfffffe3fcf1b2b70, rbp = 0 --- ] Jan 29 11:24:12 ts-prv src_dev_log@ts Buffering: S13.d06 [[2-1] FATAL: remaining connection slots are reserved for non-replication superuser connections panic: vm_radix_insert: key 46f is already present cpuid = 0 ] Jan 29 11:24:12 ts-prv src_dev_log@ts Buffering: S13.d06 [KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe3fcf1b2820 ] Jan 29 11:24:12 ts-prv src_dev_log@ts Buffering: S13.d06 [kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf1b28d0 panic() at panic+0x155/frame 0xfffffe3fcf1b2950 ] Jan 29 11:24:12 ts-prv src_dev_log@ts Buffering: S13.d06 [vm_radix_insert() at vm_radix_insert+0x2ed/frame 0xfffffe3fcf1b29b0 vm_page_cache() at vm_page_cache+0x121/frame 0xfffffe3fcf1b29f0 ] Jan 29 11:24:12 ts-prv src_dev_log@ts Buffering: S13.d06 [vm_pageout() at vm_pageout+0x8f7/frame 0xfffffe3fcf1b2a70 fork_exit() at fork_exit+0x9a/frame 0xfffffe3fcf1b2ab0 ] Jan 29 11:24:12 ts-prv src_dev_log@ts Buffering: S13.d06 [fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe3fcf1b2ab0 --- trap 0, rip = 0, rsp = 0xfffffe3fcf1b2b70, rbp = 0 --- ] On this last one, it appears that postgres reached its connection limit at the time of the panic. -- You are receiving this mail because: You are the assignee for the bug.