From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 25 15:53:25 2006 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 29B5E16A407 for ; Wed, 25 Oct 2006 15:53:25 +0000 (UTC) (envelope-from micahjon@ywave.com) Received: from relay0.av-mx.com (relay0.av-mx.com [137.118.16.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8807043D49 for ; Wed, 25 Oct 2006 15:53:24 +0000 (GMT) (envelope-from micahjon@ywave.com) X-Virus-Scan-Time: 0 Received: from [137.118.16.54] (HELO mx4.av-mx.com) by relay0.av-mx.com (CommuniGate Pro SMTP 4.2.10) with SMTP id 443318183 for freebsd-hackers@freebsd.org; Wed, 25 Oct 2006 11:52:13 -0400 Received: (qmail 16462 invoked by uid 504); 25 Oct 2006 15:52:12 -0000 Received: from dsl28213.ywave.com (HELO ?192.168.1.65?) (micahjon@ywave.com@216.227.115.213) by 0 with SMTP; 25 Oct 2006 15:52:12 -0000 X-CLIENT-IP: 216.227.115.213 X-CLIENT-HOST: dsl28213.ywave.com Message-ID: <453F882B.2040009@ywave.com> Date: Wed, 25 Oct 2006 08:52:11 -0700 From: Micah User-Agent: Thunderbird 1.5.0.7 (X11/20061019) MIME-Version: 1.0 To: FreeBSD Hackers Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: System panic under load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Oct 2006 15:53:25 -0000 Can anyone help me in deciphering what this crashdump is telling me? My i386 6.1p3 system is panicking under heavy IO load and I'm trying to isolate the cause. I've never debugged a crashdump before, so if there's more information I can milk from the core file, let me know. Thanks, Micah trisha# kgdb kernel.debug /home/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0x27 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0593d34 stack pointer = 0x28:0xf02ca89c frame pointer = 0x28:0xf02ca8bc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 24805 (perl5.8.8) trap number = 12 panic: page fault Uptime: 13h16m1s Dumping 1534 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1534MB (392672 pages) 1518 1502 1486 1470 1454 1438 1422 1406 1390 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc0535a3f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402 #2 0xc0535d66 in panic (fmt=0xc0711e06 "%s") at /usr/src/sys/kern/kern_shutdown.c:558 #3 0xc06ecf2c in trap_fatal (frame=0xf02ca85c, eva=0) at /usr/src/sys/i386/i386/trap.c:836 #4 0xc06ecc32 in trap_pfault (frame=0xf02ca85c, usermode=0, eva=39) at /usr/src/sys/i386/i386/trap.c:744 #5 0xc06ec7fd in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 4, tf_esi = 4, tf_ebp = -265508676, tf_isp = -265508728, tf_ebx = -1, tf_edx = -980369408, tf_ecx = -979030016, tf_eax = 5347783, tf_trapno = 12, tf_err = 0, tf_eip = -1067893452, tf_cs = 32, tf_eflags = 66182, tf_esp = -979030016, tf_ss = 5347783}) at /usr/src/sys/i386/i386/trap.c:434 #6 0xc06d9e2a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc0593d34 in vfs_hash_get (mp=0xc5a53000, hash=5347783, flags=2, td=0xc5ef9d80, vpp=0xf02ca99c, fn=0, arg=0x0) at /usr/src/sys/kern/vfs_hash.c:73 #8 0xc067af59 in ffs_vget (mp=0xc5a53000, ino=5347783, flags=2, vpp=0xf02ca99c) at pcpu.h:162 #9 0xc0683353 in ufs_lookup (ap=0xf02caa40) at /usr/src/sys/ufs/ufs/ufs_lookup.c:572 #10 0xc0700f03 in VOP_CACHEDLOOKUP_APV (vop=0x5199c7, a=0xc590c000) at vnode_if.c:150 #11 0xc059006a in vfs_cache_lookup (ap=0x5199c7) at vnode_if.h:82 #12 0xc0700e78 in VOP_LOOKUP_APV (vop=0xc0764f20, a=0xf02caaec) at vnode_if.c:99 #13 0xc059547b in lookup (ndp=0xf02cabf4) at vnode_if.h:56 #14 0xc0594c08 in namei (ndp=0xf02cabf4) at /usr/src/sys/kern/vfs_lookup.c:203 #15 0xc05a672d in kern_stat (td=0xc5ef9d80, path=0xc590c000 "°\033\210Ê B\215É0£ÓÊ\020±ZÉ\020ñ\211Æ0\003\237ÊPU0Ê", pathseg=3314597888, sbp=0x5199c7) at /usr/src/sys/kern/vfs_syscalls.c:2076 #16 0xc05a66bf in stat (td=0x5199c7, uap=0xf02cad04) at /usr/src/sys/kern/vfs_syscalls.c:2061 #17 0xc06ed302 in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 1209537912, tf_esi = 140130588, tf_ebp = -1077941096, tf_isp = -265507484, tf_ebx = 1209530852, tf_edx = 68, tf_ecx = 0, tf_eax = 188, tf_trapno = 12, tf_err = 2, tf_eip = 1210540343, tf_cs = 51, tf_eflags = 582, tf_esp = -1077941140, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:981 #18 0xc06d9e7f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 ---Type to continue, or q to quit--- #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?)