From owner-freebsd-bugs@FreeBSD.ORG Thu Jun 14 20:58:18 2007 Return-Path: X-Original-To: freebsd-bugs@freebsd.org Delivered-To: freebsd-bugs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6C57D16A47A for ; Thu, 14 Jun 2007 20:58:18 +0000 (UTC) (envelope-from dan@more.net) Received: from jane.spg.more.net (jane.spg.more.net [207.160.133.140]) by mx1.freebsd.org (Postfix) with ESMTP id DDAF313C4D3 for ; Thu, 14 Jun 2007 20:58:16 +0000 (UTC) (envelope-from dan@more.net) Received: from [127.0.0.1] (localhost [127.0.0.1]) by jane.spg.more.net (Postfix) with ESMTP id 39A9A9C51F for ; Thu, 14 Jun 2007 15:35:06 -0500 (CDT) From: Dan D Niles To: freebsd-bugs@freebsd.org Content-Type: text/plain Date: Thu, 14 Jun 2007 15:35:06 -0500 Message-Id: <1181853306.30559.82.camel@jane.spg.more.net> Mime-Version: 1.0 X-Mailer: Evolution 2.10.1 Content-Transfer-Encoding: 7bit Subject: Kernel Panic (Fatal trap 12: page fault while in kernel mode) X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jun 2007 20:58:18 -0000 I'm not sure if this is a new issue or an existing one. I see some similar issues, but I can't tell if they are the same. If this is a new issue, I can submit a PR. I have 4 identical Dell 2650 servers. I upgraded one of them from 6.1 to 6.2 and it started crashing. I installed it from scratch thinking something got hosed in the upgrade, but it still crashed. I swapped disks with one of the other 6.1 machines to rule out hardware. FreeBSD 6.2 is still crashing on different hardware (except the OS disk). I enabled crash dumps. Here is some output from kgdb: # kgdb kernel.debug /var/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x104 fault code = supervisor read, page not present instruction pointer = 0x20:0xc066c761 stack pointer = 0x28:0xe4f8ec90 frame pointer = 0x28:0xe4f8ec9c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 5 (thread taskq) trap number = 12 panic: page fault cpuid = 1 Uptime: 1d6h4m40s Dumping 2047 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 2047MB (524016 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc067553a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0675861 in panic (fmt=0xc08e4721 "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc088e2dc in trap_fatal (frame=0xe4f8ec50, eva=260) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc088da4e in trap (frame= {tf_fs = -969015288, tf_es = -942342104, tf_ds = -453509080, tf_edi = -969003008, tf_esi = 4, tf_ebp = -453448548, tf_isp = -453448580, tf_ebx = -955327240, tf_edx = 6, tf_ecx = 0, tf_eax = 1, tf_trapno = 12, tf_err = 0, tf_eip = -1067006111, tf_cs = 32, tf_eflags = 65538, tf_esp = -955736776, tf_ss = 4}) at /usr/src/sys/i386/i386/trap.c:270 #5 0xc0879d8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc066c761 in _mtx_lock_sleep (m=0xc70edcf8, tid=3325964288, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:546 #7 0xc06bbac6 in unp_gc (arg=0x0, pending=2) at /usr/src/sys/kern/uipc_usrreq.c:1714 #8 0xc06961e3 in taskqueue_run (queue=0xc64b0200) at /usr/src/sys/kern/subr_taskqueue.c:257 #9 0xc06966c6 in taskqueue_thread_loop (arg=0x1) at /usr/src/sys/kern/subr_taskqueue.c:376 #10 0xc065ec7d in fork_exit (callout=0xc0696634 , arg=0xc09e3ca8, frame=0xe4f8ed38) at /usr/src/sys/kern/kern_fork.c:821 ---Type to continue, or q to quit--- #11 0xc0879dec in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 Is the Undefined symbol "ps_pglobal_lookup" at the start of the kgdb output a problem? I'd like to upgrade all 4 servers to 6.2, but I need to resolve this first. What can I do to help debug this? I have the vmcore.0 file, and I can crash the box within a day or two. Thanks, Dan PS Please be sure to cc me, I'm not on the list.