From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 7 07:47:21 2005 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EC45C16A41F; Wed, 7 Dec 2005 07:47:20 +0000 (GMT) (envelope-from yuri@irfu.se) Received: from colibri.its.uu.se (colibri.its.uu.se [130.238.4.154]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5193B43D5E; Wed, 7 Dec 2005 07:47:16 +0000 (GMT) (envelope-from yuri@irfu.se) Received: by colibri.its.uu.se (Postfix, from userid 211) id F1028F76; Wed, 7 Dec 2005 08:47:14 +0100 (NFT) Received: from colibri.its.uu.se(127.0.0.1) by colibri.its.uu.se via virus-scan id s14345; Wed, 7 Dec 05 08:47:07 +0100 Received: from hq.irfu.se (hq.irfu.se [130.238.30.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by colibri.its.uu.se (Postfix) with ESMTP id BF8DBF73; Wed, 7 Dec 2005 08:47:06 +0100 (NFT) Received: from ice.irfu.se (ice.irfu.se [130.238.30.157]) (authenticated bits=0) by hq.irfu.se (8.13.4/8.13.4) with ESMTP id jB77l56H018503 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO); Wed, 7 Dec 2005 08:47:06 +0100 (CET) (envelope-from yuri@irfu.se) From: Yuri Khotyaintsev Organization: Swedish Institute of Space Physics To: John Baldwin Date: Wed, 7 Dec 2005 08:47:04 +0100 User-Agent: KMail/1.8.3 References: <200512021100.03167.yuri@irfu.se> <200512020854.20959.jhb@freebsd.org> In-Reply-To: <200512020854.20959.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200512070847.04980.yuri@irfu.se> X-Spam-Status: No, score=-1.4 required=7.0 tests=ALL_TRUSTED autolearn=failed version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on hq.irfu.se X-Mailman-Approved-At: Wed, 07 Dec 2005 12:35:54 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Fatal trap 12: page fault while in kernel mode X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Dec 2005 07:47:21 -0000 On Friday 02 December 2005 14.54, John Baldwin wrote: > On Friday 02 December 2005 05:00 am, Yuri Khotyaintsev wrote: > > I have the following panic occurring several times a week. The machine = is > > an NFS server, and it usually panics early in the morning, when first > > people try to access it. After reboot it may work OK for 1-2 days, and > > then panics again. I have tried changing memory and replacing disk which > > was exported via NFS, but nothing helped :( > > > > Any suggestion on how to fix this panic will be very much appreciated ! > > This panic (in propagate_priority) is usually caused when a thread goes to > sleep while holding a mutex (which is forbidden). If you enable INVARIAN= TS > and/or WITNESS you should get a better panic, and with WITNESS you will > even be warned when a thread goes to sleep while holding a mutex. Howeve= r, > these options do introduce considerable execution overhead, and sometimes > that overhead changes the timing enough to hide the race. :( Here are the two panics which I got with INVARIANTS and WITNESS enabled. # kgdb /usr/obj/usr/src/sys/HEM.DEBUG/kernel.debug vmcore.8=20 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:= =20 Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Memory modified after free 0xc4759e00(508) val=3D0 @ 0xc4759e00 panic: Most recently used by UFS dirhash Uptime: 11h8m36s Dumping 511 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335= =20 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) where #0 doadump () at pcpu.h:165 #1 0xc050fd4f in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:3= 99 #2 0xc0510043 in panic (fmt=3D0xc06dccbb "Most recently used by %s\n") at /usr/src/sys/kern/kern_shutdown.c:555 #3 0xc0648ccf in mtrash_ctor (mem=3D0xc4759e00, size=3D0, arg=3D0x0, flags= =3D2) at /usr/src/sys/vm/uma_dbg.c:137 #4 0xc06469c1 in uma_zalloc_arg (zone=3D0xc104d980, udata=3D0x0, flags=3D2) at /usr/src/sys/vm/uma_core.c:1850 #5 0xc05043cd in malloc (size=3D400, mtp=3D0xc06fb700, flags=3D2) at uma.h= :275 #6 0xc063fba9 in ufs_readdir (ap=3D0xd56eaaec) at /usr/src/sys/ufs/ufs/ufs_vnops.c:1846 #7 0xc06a61cc in VOP_READDIR_APV (vop=3D0x0, a=3D0xd56eaaec) at vnode_if.c= :1427 #8 0xc0607716 in nfsrv_readdir (nfsd=3D0xc4368c00, slp=3D0x0, td=3D0xc3326= 780,=20 mrq=3D0xd56eac80) at vnode_if.h:746 #9 0xc060fa5b in nfssvc_nfsd (td=3D0x0) at /usr/src/sys/nfsserver/nfs_syscalls.c:472 #10 0xc060f280 in nfssvc (td=3D0xc3326780, uap=3D0xd56ead04) at /usr/src/sys/nfsserver/nfs_syscalls.c:181 #11 0xc069b6b0 in syscall (frame=3D =2D--Type to continue, or q to quit--- {tf_fs =3D 59, tf_es =3D 59, tf_ds =3D 59, tf_edi =3D 0, tf_esi =3D 0= , tf_ebp =3D=20 =2D1077941464, tf_isp =3D -714166940, tf_ebx =3D 0, tf_edx =3D -1077936144,= tf_ecx =3D=20 1, tf_eax =3D 155, tf_trapno =3D 12, tf_err =3D 2, tf_eip =3D 671852067, tf= _cs =3D 51,=20 tf_eflags =3D 582, tf_esp =3D -1077941492, tf_ss =3D 59})=20 at /usr/src/sys/i386/i386/trap.c:981 #12 0xc068947f in Xint0x80_syscall ()=20 at /usr/src/sys/i386/i386/exception.s:200 #13 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) quit # kgdb /usr/obj/usr/src/sys/HEM.DEBUG/kernel.debug vmcore.9 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:= =20 Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Memory modified after free 0xc5172800(508) val=3D0 @ 0xc5172800 panic: Most recently used by UFS dirhash Uptime: 1d1h7m17s Dumping 511 MB (2 chunks) chunk 0: 1MB (160 pages) ... ok chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335= =20 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) where #0 doadump () at pcpu.h:165 #1 0xc050fd4f in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:3= 99 #2 0xc0510043 in panic (fmt=3D0xc06dccbb "Most recently used by %s\n") at /usr/src/sys/kern/kern_shutdown.c:555 #3 0xc0648ccf in mtrash_ctor (mem=3D0xc5172800, size=3D0, arg=3D0x0, flags= =3D257) at /usr/src/sys/vm/uma_dbg.c:137 #4 0xc06469c1 in uma_zalloc_arg (zone=3D0xc104d980, udata=3D0x0, flags=3D2= 57) at /usr/src/sys/vm/uma_core.c:1850 #5 0xc05043cd in malloc (size=3D368, mtp=3D0xc070eb60, flags=3D257) at uma= =2Eh:275 #6 0xc063729b in ufsdirhash_build (ip=3D0xc55664a4) at /usr/src/sys/ufs/ufs/ufs_dirhash.c:184 #7 0xc0639441 in ufs_lookup (ap=3D0xd57c283c) at /usr/src/sys/ufs/ufs/ufs_lookup.c:192 #8 0xc06a4e0a in VOP_CACHEDLOOKUP_APV (vop=3D0x0, a=3D0xd57c283c) at vnode_if.c:150 #9 0xc0565e3b in vfs_cache_lookup (ap=3D0x0) at vnode_if.h:82 #10 0xc06a4d2f in VOP_LOOKUP_APV (vop=3D0xc070eee0, a=3D0xd57c28e4) at vnode_if.c:99 #11 0xc056a8d0 in lookup (ndp=3D0xd57c2bec) at vnode_if.h:56 =2D--Type to continue, or q to quit--- #12 0xc060df58 in nfs_namei (ndp=3D0xd57c2bec, fhp=3D0x0, len=3D0, slp=3D0x= 0, nam=3D0x0,=20 mdp=3D0xd57c2a04, dposp=3D0xd57c2a08, retdirp=3D0xd57c29f0, v3=3D8,=20 retdirattrp=3D0x0, retdirattr_retp=3D0x0, td=3D0xc350a780, pubflag=3D0) at /usr/src/sys/nfsserver/nfs_srvsubs.c:780 #13 0xc05fd284 in nfsrv_lookup (nfsd=3D0xc5764100, slp=3D0x0, td=3D0xc350a7= 80,=20 mrq=3D0xd57c2c80) at /usr/src/sys/nfsserver/nfs_serv.c:517 #14 0xc060fa5b in nfssvc_nfsd (td=3D0x0) at /usr/src/sys/nfsserver/nfs_syscalls.c:472 #15 0xc060f280 in nfssvc (td=3D0xc350a780, uap=3D0xd57c2d04) at /usr/src/sys/nfsserver/nfs_syscalls.c:181 #16 0xc069b6b0 in syscall (frame=3D {tf_fs =3D 59, tf_es =3D 59, tf_ds =3D 59, tf_edi =3D 0, tf_esi =3D 0= , tf_ebp =3D=20 =2D1077941464, tf_isp =3D -713282204, tf_ebx =3D 0, tf_edx =3D -1077936144,= tf_ecx =3D=20 1, tf_eax =3D 155, tf_trapno =3D 12, tf_err =3D 2, tf_eip =3D 671852067, tf= _cs =3D 51,=20 tf_eflags =3D 582, tf_esp =3D -1077941492, tf_ss =3D 59})=20 at /usr/src/sys/i386/i386/trap.c:981 #17 0xc068947f in Xint0x80_syscall ()=20 at /usr/src/sys/i386/i386/exception.s:200 #18 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) exit Undefined command: "exit". Try "help". (kgdb) quit =2D-=20 Dr. Yuri Khotyaintsev Institutet f=F6r rymdfysik (IRF), Uppsala