From owner-freebsd-current@FreeBSD.ORG Fri Jan 4 01:56:24 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5EB9B16A417 for ; Fri, 4 Jan 2008 01:56:24 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id 76BC513C45B for ; Fri, 4 Jan 2008 01:56:23 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (unknown [202.108.54.204]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTP id B73DF28481 for ; Fri, 4 Jan 2008 09:56:21 +0800 (CST) Received: from localhost (unknown [202.108.54.204]) by tarsier.geekcn.org (Postfix) with ESMTP id 5B83EEDB284; Fri, 4 Jan 2008 09:56:21 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([202.108.54.204]) by localhost (mail.geekcn.org [202.108.54.204]) (amavisd-new, port 10024) with ESMTP id NMtfgMtCh+ep; Fri, 4 Jan 2008 09:56:19 +0800 (CST) Received: from charlie.delphij.net (71.5.7.139.ptr.us.xo.net [71.5.7.139]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTP id 169D3EDB32A; Fri, 4 Jan 2008 09:56:00 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:subject:x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=EG/HWp6k3mn6GuOaYy6GKuJ+68q2AAYoCUahQeoT3AwMOYlK/8tnzLC2+j53n/J1P PL0h+NxMXWSOBjVFk+OPA== Message-ID: <477D922F.2040308@delphij.net> Date: Thu, 03 Jan 2008 17:55:59 -0800 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.9 (X11/20071125) MIME-Version: 1.0 To: FreeBSD Current X-Enigmail-Version: 0.95.5 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Strange kernel trap 12 with vm_page_splay() on FreeBSD/i386 SMP 7.0-RC1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jan 2008 01:56:24 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Recently I have encountered a strange kernel trap 12, which always end up at vm_page_splay() like this, the fault va and IP vary from time to time: Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0xfff9a4b5 fault code = supervisor read, page not present instruction pointer = 0x20:0xc073eec7 stack pointer = 0x28:0xe66fa94c frame pointer = 0x28:0xe66fa9a4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1123 (mysqld) trap number = 12 panic: page fault cpuid = 3 Uptime: 10m6s Physical memory: 1015 MB Dumping 130 MB: 115 99 83 67 51 35 19 3 And the backtrace was: #0 doadump () at pcpu.h:195 #1 0xc0566b37 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0566df9 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc0777eec in trap_fatal (frame=0xe66fa90c, eva=4294550709) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc0778150 in trap_pfault (frame=0xe66fa90c, usermode=0, eva=4294550709) at /usr/src/sys/i386/i386/trap.c:812 #5 0xc0778aa2 in trap (frame=0xe66fa90c) at /usr/src/sys/i386/i386/trap.c:490 #6 0xc075f47b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc073eec7 in vm_page_splay (pindex=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/vm/vm_page.c:576 #8 0xc073efad in vm_page_lookup (object=0xc4513a2c, pindex=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/vm/vm_page.c:759 #9 0xc05ca4f6 in allocbuf (bp=0xd7d20f20, size=16384) at /usr/src/sys/kern/vfs_bio.c:2884 #10 0xc05cddcd in getblk (vp=0xc44a5330, blkno=4379, size=16384, slpflag=0, slptimeo=0, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/vfs_bio.c:2662 #11 0xc05d08d7 in cluster_read (vp=0xc44a5330, filesize=104590932, lblkno=4380, size=16384, cred=0x0, totread=5240, seqcount=0, bpp=0xe66fab80) at /usr/src/sys/kern/vfs_cluster.c:118 #12 0xc0717d20 in ffs_read (ap=0xe66fabc8) at /usr/src/sys/ufs/ffs/ffs_vnops.c:511 #13 0xc0782c72 in VOP_READ_APV (vop=0xc07f7140, a=0xe66fabc8) at vnode_if.c:637 #14 0xc05ee5f4 in vn_read (fp=0xc3e09c60, uio=0xe66fac60, active_cred=0xc4142500, flags=0, td=0xc41d9880) at vnode_if.h:344 #15 0xc059b5e6 in dofileread (td=0xc41d9880, fd=106, fp=0xc3e09c60, auio=0xe66fac60, offset=-1, flags=0) at file.h:242 #16 0xc059b958 in kern_readv (td=0xc41d9880, fd=106, auio=0xe66fac60) at /usr/src/sys/kern/sys_generic.c:192 #17 0xc059ba3f in read (td=0xc41d9880, uap=0xe66facfc) at /usr/src/sys/kern/sys_generic.c:108 #18 0xc0778489 in syscall (frame=0xe66fad38) at /usr/src/sys/i386/i386/trap.c:1035 #19 0xc075f4e0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196 #20 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) Another one: #0 doadump () at pcpu.h:195 #1 0xc0566b37 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc0566df9 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xc0777eec in trap_fatal (frame=0xe68fd90c, eva=9437986) at /usr/src/sys/i386/i386/trap.c:899 #4 0xc0778150 in trap_pfault (frame=0xe68fd90c, usermode=0, eva=9437986) at /usr/src/sys/i386/i386/trap.c:812 #5 0xc0778aa2 in trap (frame=0xe68fd90c) at /usr/src/sys/i386/i386/trap.c:490 #6 0xc075f47b in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc073ef09 in vm_page_splay (pindex=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/vm/vm_page.c:590 #8 0xc073efad in vm_page_lookup (object=0xc4623934, pindex=Unhandled dwarf expression opcode 0x93 ) at /usr/src/sys/vm/vm_page.c:759 #9 0xc05ca4f6 in allocbuf (bp=0xd7d0b640, size=16384) at /usr/src/sys/kern/vfs_bio.c:2884 #10 0xc05cddcd in getblk (vp=0xc4827aa0, blkno=63, size=16384, slpflag=0, slptimeo=0, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/vfs_bio.c:2662 #11 0xc05d08d7 in cluster_read (vp=0xc4827aa0, filesize=3393192, lblkno=64, size=16384, cred=0x0, totread=15352, seqcount=0, bpp=0xe68fdb80) at /usr/src/sys/kern/vfs_cluster.c:118 #12 0xc0717d20 in ffs_read (ap=0xe68fdbc8) at /usr/src/sys/ufs/ffs/ffs_vnops.c:511 #13 0xc0782c72 in VOP_READ_APV (vop=0xc07f7140, a=0xe68fdbc8) at vnode_if.c:637 #14 0xc05ee5f4 in vn_read (fp=0xc45941b0, uio=0xe68fdc60, active_cred=0xc3e35000, flags=0, td=0xc4f05660) at vnode_if.h:344 #15 0xc059b5e6 in dofileread (td=0xc4f05660, fd=176, fp=0xc45941b0, auio=0xe68fdc60, offset=-1, flags=0) at file.h:242 #16 0xc059b958 in kern_readv (td=0xc4f05660, fd=176, auio=0xe68fdc60) at /usr/src/sys/kern/sys_generic.c:192 #17 0xc059ba3f in read (td=0xc4f05660, uap=0xe68fdcfc) at /usr/src/sys/kern/sys_generic.c:108 #18 0xc0778489 in syscall (frame=0xe68fdd38) at /usr/src/sys/i386/i386/trap.c:1035 #19 0xc075f4e0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196 #20 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) Because this is observed on two servers, it smells like a software issue, but I have not yet completely ruled hardware factor out. Anyone has some debugging hints for this? Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHfZIvi+vbBBjt66ARAqqnAJ9QgJnKER7GmxmehDUeG4oUI+9JBgCfdjrv 0RGvBHUAUUp8si/Mc9Z6qck= =DD59 -----END PGP SIGNATURE-----