From owner-freebsd-hackers@FreeBSD.ORG Tue Jan 23 16:40:29 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C642316A402 for ; Tue, 23 Jan 2007 16:40:29 +0000 (UTC) (envelope-from micahjon@ywave.com) Received: from relay0.av-mx.com (relay0.av-mx.com [137.118.16.125]) by mx1.freebsd.org (Postfix) with ESMTP id 5B56E13C4BD for ; Tue, 23 Jan 2007 16:40:29 +0000 (UTC) (envelope-from micahjon@ywave.com) X-Virus-Scan-Time: 0 Received: from mx0.av-mx.com ([137.118.16.61] verified) by relay0.av-mx.com (CommuniGate Pro SMTP 5.0.13) with SMTP id 567893306 for freebsd-hackers@freebsd.org; Tue, 23 Jan 2007 11:40:24 -0500 Received: (qmail 16050 invoked from network); 23 Jan 2007 16:40:23 -0000 Received: from dsl10219.ywave.com (HELO ?192.168.1.66?) (micahjon@ywave.com@12.178.97.219) by 0 with SMTP; 23 Jan 2007 16:40:23 -0000 X-CLIENT-IP: 12.178.97.219 X-CLIENT-HOST: dsl10219.ywave.com Message-ID: <45B63A79.3030805@ywave.com> Date: Tue, 23 Jan 2007 08:40:25 -0800 From: Micah User-Agent: Thunderbird 1.5.0.9 (X11/20070120) MIME-Version: 1.0 To: FreeBSD Hackers Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Subject: Page fault in vfs_hash_get on 6.2 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2007 16:40:30 -0000 Back in October 2006, I was having the same problem in 6.1 (original thread at: http://lists.freebsd.org/pipermail/freebsd-hackers/2006-October/018393.html) The update to 6.2 decreased the frequency of the panics, but I still get the panic periodically - since October I've had 6-8 panics judging by my vmcore count. trisha# uname -a FreeBSD trisha.lan 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Sat Jan 20 16:18:57 PST 2007 root@trisha.lan:/usr/obj/usr/src/sys/TRISHA i386 Here are the last two dumps with some prints of the interesting vars (I have more dumps, but these are the only two for 6.2-release). Let me know if I can provide anything else. trisha# kgdb /usr/obj/usr/src/sys/TRISHA/kernel.debug vmcore.13 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xd92c1358 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0595304 stack pointer = 0x28:0xf039c89c frame pointer = 0x28:0xf039c8bc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 15173 (kphotoalbum) trap number = 12 panic: page fault Uptime: 2d4h53m29s Dumping 1534 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1534MB (392672 pages) 1518 1502 1486 1470 1454 1438 1422 1406 1390 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 158 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 142 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 126 110 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 94 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 78 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 62 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 46 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 30 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) 14 (CTRL-C to abort) (CTRL-C to abort) (CTRL-C to abort) #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc0535f54 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0536286 in panic (fmt=0xc071898d "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc06f34bc in trap_fatal (frame=0xf039c85c, eva=0) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc06f31c2 in trap_pfault (frame=0xf039c85c, usermode=0, eva=3643544408) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc06f2d8d in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 4, tf_esi = 4, tf_ebp = -264648516, tf_isp = -264648568, tf_ebx = -651422928, tf_edx = -980348928, tf_ecx = -978780160, tf_eax = 8055459, tf_trapno = 12, tf_err = 0, tf_eip = -1067887868, tf_cs = 32, tf_eflags = 2163334, tf_esp = -978780160, tf_ss = 8055459}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc06df32a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc0595304 in vfs_hash_get (mp=0xc5a90000, hash=8055459, flags=2, td=0xc6552600, vpp=0xf039c99c, fn=0, arg=0x0) at /usr/src/sys/kern/vfs_hash.c:73 #8 0xc067f379 in ffs_vget (mp=0xc5a90000, ino=8055459, flags=2, vpp=0xf039c99c) at pcpu.h:162 #9 0xc06879f3 in ufs_lookup (ap=0xf039ca40) at /usr/src/sys/ufs/ufs/ufs_lookup.c:572 #10 0xc0707993 in VOP_CACHEDLOOKUP_APV (vop=0x7aeaa3, a=0xc5911000) at vnode_if.c:150 #11 0xc05913ea in vfs_cache_lookup (ap=0x7aeaa3) at vnode_if.h:82 #12 0xc0707908 in VOP_LOOKUP_APV (vop=0xc076c500, a=0xf039caec) at vnode_if.c:99 #13 0xc0596a3b in lookup (ndp=0xf039cb94) at vnode_if.h:56 #14 0xc05961d8 in namei (ndp=0xf039cb94) at /usr/src/sys/kern/vfs_lookup.c:211 #15 0xc05a86cf in kern_lstat (td=0xc6552600, path=0xc5911000 "", pathseg=3314618368, sbp=0x7aeaa3) at /usr/src/sys/kern/vfs_syscalls.c:2143 #16 0xc05a864f in lstat (td=0x7aeaa3, uap=0xf039cd04) at /usr/src/sys/kern/vfs_syscalls.c:2126 #17 0xc06f3892 in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 139441408, tf_esi = -1077945056, tf_ebp = -1077945816, tf_isp = -264647324, tf_ebx = 1230067672, tf_edx = 69, tf_ecx = 142276000, tf_eax = 190, tf_trapno = 0, tf_err = 2, tf_eip = 1233487479, tf_cs = 51, tf_eflags = 2097794, tf_esp = -1077945892, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 #18 0xc06df37f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) frame 7 #7 0xc0595304 in vfs_hash_get (mp=0xc5a90000, hash=8055459, flags=2, td=0xc6552600, vpp=0xf039c99c, fn=0, arg=0x0) at /usr/src/sys/kern/vfs_hash.c:73 73 if (vp->v_hash != hash) (kgdb) p mp $1 = (struct mount *) 0xc5a90000 (kgdb) p *mp $2 = {mnt_list = {tqe_next = 0x0, tqe_prev = 0xc5a90298}, mnt_op = 0xc076bc60, mnt_vfc = 0xc076bca0, mnt_vnodecovered = 0xc5ad0aa0, mnt_syncer = 0xc5ad4cc0, mnt_nvnodelist = {tqh_first = 0xc5ad4dd0, tqh_last = 0xcaabc014}, mnt_lock = {lk_interlock = 0xc077f11c, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_prio = 80, lk_wmesg = 0xc072d526 "vfslock", lk_timo = 0, lk_lockholder = 0xffffffff, lk_newlock = 0x0}, mnt_mtx = {mtx_object = { lo_class = 0xc075a8a4, lo_name = 0xc072d515 "struct mount mtx", lo_type = 0xc072d515 "struct mount mtx", lo_flags = 196608, lo_list = { tqe_next = 0x0, tqe_prev = 0x0}, lo_witness = 0x0}, mtx_lock = 4, mtx_recurse = 0}, mnt_writeopcount = 0, mnt_flag = 2101248, mnt_opt = 0xc5a498a0, mnt_optnew = 0x0, mnt_kern_flag = 536870912, mnt_maxsymlinklen = 120, mnt_stat = {f_version = 537068824, f_type = 5, f_flags = 2101248, f_bsize = 2048, f_iosize = 16384, f_blocks = 47731967, f_bfree = 11478076, f_bavail = 7659519, f_files = 12341246, f_ffree = 11381805, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, f_fsid = {val = {1161696182, 824004859}}, f_charspare = '\0' , f_fstypename = "ufs", '\0' , f_mntfromname = "/dev/ad4s1h", '\0' , f_mntonname = "/home", '\0' }, mnt_cred = 0xc5a24c00, mnt_data = 0xc5a6b900, mnt_time = 0, mnt_iosize_max = 131072, mnt_export = 0x0, mnt_mntlabel = 0x0, mnt_fslabel = 0x0, mnt_nvnodelistsize = 81992, mnt_hashseed = 1412285663, mnt_markercnt = 0, mnt_holdcnt = 0, mnt_holdcntwaiters = 0, mnt_secondary_writes = 0, mnt_secondary_accwrites = 2436404, mnt_ref = 81992, mnt_gen = 1} (kgdb) p vp $3 = (struct vnode *) 0xd92c1330 (kgdb) p *vp Cannot access memory at address 0xd92c1330 (kgdb) trisha# kgdb /usr/obj/usr/src/sys/TRISHA/kernel.debug vmcore.14 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0xce7b0df8 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0595304 stack pointer = 0x28:0xf03ab89c frame pointer = 0x28:0xf03ab8bc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2897 (find) trap number = 12 panic: page fault Uptime: 5h17m47s Dumping 1534 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1534MB (392672 pages) 1518 1502 1486 1470 1454 1438 1422 1406 1390 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc0535f54 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0536286 in panic (fmt=0xc071898d "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc06f34bc in trap_fatal (frame=0xf03ab85c, eva=0) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc06f31c2 in trap_pfault (frame=0xf03ab85c, usermode=0, eva=3464171000) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc06f2d8d in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 4, tf_esi = 4, tf_ebp = -264587076, tf_isp = -264587128, tf_ebx = -830796336, tf_edx = -980348928, tf_ecx = -978780160, tf_eax = 4027855, tf_trapno = 12, tf_err = 0, tf_eip = -1067887868, tf_cs = 32, tf_eflags = 66178, tf_esp = -978780160, tf_ss = 4027855}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc06df32a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc0595304 in vfs_hash_get (mp=0xc5a90000, hash=4027855, flags=2, td=0xc64a3d80, vpp=0xf03ab99c, fn=0, arg=0x0) at /usr/src/sys/kern/vfs_hash.c:73 #8 0xc067f379 in ffs_vget (mp=0xc5a90000, ino=4027855, flags=2, vpp=0xf03ab99c) at pcpu.h:162 #9 0xc06879f3 in ufs_lookup (ap=0xf03aba40) at /usr/src/sys/ufs/ufs/ufs_lookup.c:572 #10 0xc0707993 in VOP_CACHEDLOOKUP_APV (vop=0x3d75cf, a=0xc5911000) at vnode_if.c:150 #11 0xc05913ea in vfs_cache_lookup (ap=0x3d75cf) at vnode_if.h:82 #12 0xc0707908 in VOP_LOOKUP_APV (vop=0xc076c500, a=0xf03abaec) at vnode_if.c:99 #13 0xc0596a3b in lookup (ndp=0xf03abb94) at vnode_if.h:56 #14 0xc05961d8 in namei (ndp=0xf03abb94) at /usr/src/sys/kern/vfs_lookup.c:211 #15 0xc05a86cf in kern_lstat (td=0xc64a3d80, path=0xc5911000 "@ÔLÆ0ÓLÆ ÒLÆ\020ÑLÆ°\233LÆ \232LÆ\220\231LÆÀ\034MÆ°\033MÆ \032MÆÀ¬MÆ", pathseg=3314618368, sbp=0x3d75cf) at /usr/src/sys/kern/vfs_syscalls.c:2143 #16 0xc05a864f in lstat (td=0x3d75cf, uap=0xf03abd04) at /usr/src/sys/kern/vfs_syscalls.c:2126 #17 0xc06f3892 in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 134687816, tf_esi = 134687744, tf_ebp = -1077941128, tf_isp = -264585884, tf_ebx = 1209422816, tf_edx = 134687744, tf_ecx = 134565888, tf_eax = 190, tf_trapno = 12, tf_err = 2, tf_eip = 1209300599, tf_cs = 51, tf_eflags = 582, tf_esp = -1077941284, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 ---Type to continue, or q to quit--- #18 0xc06df37f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #19 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) frame 7 #7 0xc0595304 in vfs_hash_get (mp=0xc5a90000, hash=4027855, flags=2, td=0xc64a3d80, vpp=0xf03ab99c, fn=0, arg=0x0) at /usr/src/sys/kern/vfs_hash.c:73 73 if (vp->v_hash != hash) (kgdb) p mp $1 = (struct mount *) 0xc5a90000 (kgdb) p *mp $2 = {mnt_list = {tqe_next = 0x0, tqe_prev = 0xc5a90298}, mnt_op = 0xc076bc60, mnt_vfc = 0xc076bca0, mnt_vnodecovered = 0xc5ad0aa0, mnt_syncer = 0xc5ad4cc0, mnt_nvnodelist = {tqh_first = 0xc5ad4dd0, tqh_last = 0xc7145344}, mnt_lock = {lk_interlock = 0xc077f11c, lk_flags = 0, lk_sharecount = 0, lk_waitcount = 0, lk_exclusivecount = 0, lk_prio = 80, lk_wmesg = 0xc072d526 "vfslock", lk_timo = 0, lk_lockholder = 0xffffffff, lk_newlock = 0x0}, mnt_mtx = {mtx_object = { lo_class = 0xc075a8a4, lo_name = 0xc072d515 "struct mount mtx", lo_type = 0xc072d515 "struct mount mtx", lo_flags = 196608, lo_list = { tqe_next = 0x0, tqe_prev = 0x0}, lo_witness = 0x0}, mtx_lock = 4, mtx_recurse = 0}, mnt_writeopcount = 0, mnt_flag = 2101248, mnt_opt = 0xc5a496e0, mnt_optnew = 0x0, mnt_kern_flag = 536870912, mnt_maxsymlinklen = 120, mnt_stat = {f_version = 537068824, f_type = 5, f_flags = 2101248, f_bsize = 2048, f_iosize = 16384, f_blocks = 47731967, f_bfree = 9820243, f_bavail = 6001686, f_files = 12341246, f_ffree = 11382010, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, f_fsid = {val = {1161696182, 824004859}}, f_charspare = '\0' , f_fstypename = "ufs", '\0' , f_mntfromname = "/dev/ad4s1h", '\0' , f_mntonname = "/home", '\0' }, mnt_cred = 0xc5aaed00, mnt_data = 0xc5a99e00, mnt_time = 0, mnt_iosize_max = 131072, mnt_export = 0x0, mnt_mntlabel = 0x0, mnt_fslabel = 0x0, mnt_nvnodelistsize = 10746, mnt_hashseed = 1412285663, mnt_markercnt = 0, mnt_holdcnt = 0, mnt_holdcntwaiters = 0, mnt_secondary_writes = 0, mnt_secondary_accwrites = 334543, mnt_ref = 10746, mnt_gen = 1} (kgdb) p vp $3 = (struct vnode *) 0xce7b0dd0 (kgdb) p *vp Cannot access memory at address 0xce7b0dd0 (kgdb) Thanks, Micah