From owner-freebsd-bugs@FreeBSD.ORG Wed Aug 23 01:00:36 2006 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 74C9416A4DD for ; Wed, 23 Aug 2006 01:00:36 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id D572743D5A for ; Wed, 23 Aug 2006 01:00:32 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k7N10WUD038495 for ; Wed, 23 Aug 2006 01:00:32 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k7N10WoC038494; Wed, 23 Aug 2006 01:00:32 GMT (envelope-from gnats) Resent-Date: Wed, 23 Aug 2006 01:00:32 GMT Resent-Message-Id: <200608230100.k7N10WoC038494@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Daniel Austin Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 473AC16A4DE for ; Wed, 23 Aug 2006 00:57:21 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id A4C6B43D45 for ; Wed, 23 Aug 2006 00:57:20 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k7N0vKdi003194 for ; Wed, 23 Aug 2006 00:57:20 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id k7N0vK7q003192; Wed, 23 Aug 2006 00:57:20 GMT (envelope-from nobody) Message-Id: <200608230057.k7N0vK7q003192@www.freebsd.org> Date: Wed, 23 Aug 2006 00:57:20 GMT From: Daniel Austin To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Cc: Subject: kern/102412: kernel panic in 6.1-stable during normal operation X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Aug 2006 01:00:36 -0000 >Number: 102412 >Category: kern >Synopsis: kernel panic in 6.1-stable during normal operation >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Aug 23 01:00:31 GMT 2006 >Closed-Date: >Last-Modified: >Originator: Daniel Austin >Release: 6.1-STABLE (CVS-20060822) >Organization: Kewlio.net Limited >Environment: FreeBSD 6.1-STABLE FreeBSD 6.1-STABLE #6: Tue Aug 22 21:09:55 BST 2006 dan@:/usr/obj/usr/src/sys/kewlio i386 >Description: The server is a dedicated IRC server on one of the big 4 networks. After a few hours of normal operation, the kernel panics. I compiled the kernel with WITNESS and INVARIANTS enabled after the original kernel page faults. Crash debug (with WITNESS/INVARIANTS enabled): Unread portion of the kernel message buffer: panic: mtx_lock() of destroyed mutex @ /usr/src/sys/netinet/ip_output.c:1193 Uptime: 2h51m41s Dumping 503 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc0655934 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0655bb2 in panic (fmt=0xc087b45c "mtx_lock() of destroyed mutex @ %s:%d") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc064d16e in _mtx_lock_flags (m=0xc5c21900, opts=0, file=0xc088b6d5 "/usr/src/sys/netinet/ip_output.c", line=1193) at /usr/src/sys/kern/kern_mutex.c:281 #4 0xc06f5390 in ip_ctloutput (so=0x0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193 #5 0xc0704b6f in tcp_ctloutput (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038 #6 0xc068f6f4 in sosetopt (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/kern/uipc_socket.c:1563 #7 0xc0694045 in kern_setsockopt (td=0xc4bfd480, s=1244, level=0, name=0, val=0x0, valseg=UIO_USERSPACE, valsize=0) at /usr/src/sys/kern/uipc_syscalls.c:1351 #8 0xc0693f8e in setsockopt (td=0xc4bfd480, uap=0x0) at /usr/src/sys/kern/uipc_syscalls.c:1307 #9 0xc080fdcf in syscall (frame= {tf_fs = 59, tf_es = -1078001605, tf_ds = -1078001605, tf_edi = -1077941472, tf_esi = 1244, tf_ebp = -1077941512, tf_isp = -432108188, tf_ebx = 1244, tf_edx = 1, tf_ecx = 0, tf_eax = 105, tf_trapno = 0, tf_err = 2, tf_eip = 672523411, tf_cs = 51, tf_eflags = 646, tf_esp = -1077941556, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:981 #10 0xc07fed2f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #11 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 3 #3 0xc064d16e in _mtx_lock_flags (m=0xc5c21900, opts=0, file=0xc088b6d5 "/usr/src/sys/netinet/ip_output.c", line=1193) at /usr/src/sys/kern/kern_mutex.c:281 281 KASSERT(m->mtx_lock != MTX_DESTROYED, (kgdb) f 4 #4 0xc06f5390 in ip_ctloutput (so=0x0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193 1193 INP_LOCK(inp); (kgdb) f 5 #5 0xc0704b6f in tcp_ctloutput (so=0xc57cfb20, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038 1038 error = ip_ctloutput(so, sopt); Crash debug (without WITNESS/INVARIANTS): Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x78 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0681f29 stack pointer = 0x28:0xe63e8ab8 frame pointer = 0x28:0xe63e8abc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 624 (ircd.200608220145) trap number = 12 panic: page fault Uptime: 14h17m13s Dumping 503 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc0661ad6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0661d6c in panic (fmt=0xc08ac642 "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc0861034 in trap_fatal (frame=0xe63e8a78, eva=120) at /usr/src/sys/i386/i386/trap.c:836 #4 0xc0860816 in trap (frame= {tf_fs = -432144376, tf_es = -994115544, tf_ds = -994115544, tf_edi = 0, tf_esi = -994102144, tf_ebp = -432108868, tf_isp = -432108892, tf_ebx = -994097216, tf_edx = -994097216, tf_ecx = 4, tf_eax = -994102112, tf_trapno = 12, tf_err = 0, tf_eip = -1066918103, tf_cs = 32, tf_eflags = 65543, tf_esp = -994102144, tf_ss = -432108832}) at /usr/src/sys/i386/i386/trap.c:269 #5 0xc084fa8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc0681f29 in turnstile_setowner (ts=0xc4bf47c0, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:432 #7 0xc0682220 in turnstile_wait (lock=0xc5ef6090, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:591 #8 0xc065842c in _mtx_lock_sleep (m=0xc5ef6090, tid=3300865152, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579 #9 0xc07090ec in ip_ctloutput (so=0xc4bf34a0, sopt=0xe63e8c90) at /usr/src/sys/netinet/ip_output.c:1193 #10 0xc071971f in tcp_ctloutput (so=0xc5121000, sopt=0xe63e8c90) at /usr/src/sys/netinet/tcp_usrreq.c:1038 #11 0xc069bb10 in sosetopt (so=0xc5121000, sopt=0xe63e8c90) at /usr/src/sys/kern/uipc_socket.c:1563 #12 0xc06a0dfd in kern_setsockopt (td=0xc4bf3480, s=2662, level=-994102112, name=-994102112, val=0xc4bf47c0, valseg=UIO_USERSPACE, valsize=4) at /usr/src/sys/kern/uipc_syscalls.c:1351 #13 0xc06a0d2e in setsockopt (td=0xc4bf3480, uap=0xc4bf34a0) at /usr/src/sys/kern/uipc_syscalls.c:1307 #14 0xc086134b in syscall (frame= {tf_fs = 198377531, tf_es = 223215675, tf_ds = -1078001605, tf_edi = -1077941472, tf_esi = 2662, tf_ebp = -1077941512, tf_isp = -432108188, tf_ebx = 2662, tf_edx = 1, tf_ecx = 0, tf_eax = 105, tf_trapno = 0, tf_err = 2, tf_eip = 672523411, tf_cs = 51, tf_eflags = 646, tf_esp = -1077941556, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:981 #15 0xc084fadf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #16 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 6 #6 0xc0681f29 in turnstile_setowner (ts=0xc4bf47c0, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:432 432 ts->ts_owner = owner; (kgdb) f 7 #7 0xc0682220 in turnstile_wait (lock=0xc5ef6090, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:591 591 turnstile_setowner(ts, owner); (kgdb) f 8 #8 0xc065842c in _mtx_lock_sleep (m=0xc5ef6090, tid=3300865152, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579 579 turnstile_wait(&m->mtx_object, mtx_owner(m)); (kgdb) print *m $1 = {mtx_object = {lo_class = 0xc093be44, lo_name = 0xc08d60d8 "inp", lo_type = 0xc08d3871 "tcpinp", lo_flags = 4849664, lo_list = {tqe_next = 0x0, tqe_prev = 0x0}, lo_witness = 0x0}, mtx_lock = 6, mtx_recurse = 0} >How-To-Repeat: I have not been able to duplicate this problem on another function machine, however I installed and moved the IRC server to different hardware and the same problem occurs. This happens approximately every 3 hours and seems related to the level of network traffic. The server has fxp and em-based cards with polling enabled. I am happy to work with anyone that needs any further information or assistance in diagnosing the problem. The problem started since updating from CVS recently (a CVS build from 20060623 did not cause a kernel panic, but did cause a deadlock in the ircd process) Copies of vmcores/kernels with debugging symbols have been kept. >Fix: Unsure. >Release-Note: >Audit-Trail: >Unformatted: