From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 7 00:11:09 2006 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4B65B16A403 for ; Thu, 7 Dec 2006 00:11:09 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id AD2C743CA5 for ; Thu, 7 Dec 2006 00:10:19 +0000 (GMT) (envelope-from freebsd-hackers@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Gs6qu-00016F-GN for freebsd-hackers@freebsd.org; Thu, 07 Dec 2006 01:11:04 +0100 Received: from anthonychavez.org ([166.70.126.66]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 07 Dec 2006 01:11:04 +0100 Received: from acc by anthonychavez.org with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 07 Dec 2006 01:11:04 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Anthony Chavez Date: Wed, 06 Dec 2006 17:10:43 -0700 Lines: 143 Message-ID: <871wncin0c.fsf@hephaistos.aegaeum.anthonychavez.org> References: <87odrpr48i.fsf@hephaistos.aegaeum.anthonychavez.org> <200611021712.32665.jhb@freebsd.org> <87lkmsmbt9.fsf@hephaistos.aegaeum.anthonychavez.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: anthonychavez.org User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.0.90 (gnu/linux) Cancel-Lock: sha1:QLPfLS7XUaNTZNgcWldA5rl0eJM= Sender: news Subject: Re: Panic on 6.1-RELEASE-p3 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Dec 2006 00:11:09 -0000 --=-=-= Content-Transfer-Encoding: quoted-printable Anthony Chavez writes: > John Baldwin writes: > >> On Thursday 02 November 2006 15:19, Anthony Chavez wrote: >>> freebsd-hackers: >>>=20 >>> I have attached the backtraces of 17 core dumps from one of my >>> machines. I have several deployments of this same FreeBSD version in >>> the wild, and this is the only machine exhibiting this behavior. >>>=20 >>> Initially, the cause of these panics seemed to be related to some >>> component in the FreeBSD toolchain, as they would occur when >>> attempting to install a port. net/samba3 was the one we'd usually >>> test with, but the system would panic with most other larger packages >>> as well. >>>=20 >>> However, the system is panicking at irrgeular intervals, ranging >>> between 1-3 weeks apart. This is mostly happening in the off-hours >>> when no human users are actively using the machine, but there have >>> been 1 or 2 incidences where the machine has panicked during prime >>> time. >> >> Have you ran a memory checker or other diags to check for failing hardwa= re? > > I ran sysutils/memtest twice. Both times, it failed to panic the > system. I have yet to bring the system down and boot into a "real" > memory/hardware checker, however. I usually just go with > ultimatebootcd.com when testing, but if you have other suggestions for > tools, I'd welcome your input before we schedule to do the testing. Rather than pay us to debug the problem, our client opted to simply replace the memory, and it only took them 17 more panics to move on it until it started becoming a daily issue. What follows is yet another backtrace from a panic that occurred after about 20.5 hours of uptime. I would appreciate any further assistance in addressing this problem. =2D-=20 Anthony Chavez http://anthonychavez.org/ mailto:acc@anthonychavez.org jabber:acc@jabber.anthonychavez.org mybox% kgdb /usr/obj/usr/src/sys/WITFW/kernel.debug vmcore.37 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:= Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address =3D 0x0 fault code =3D supervisor read, page not present instruction pointer =3D 0x20:0xc06809c4 stack pointer =3D 0x28:0xe9606bb4 frame pointer =3D 0x28:0xe9606bbc code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, def32 1, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 42888 (nagios) trap number =3D 12 panic: page fault Uptime: 20h27m59s Dumping 1023 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1023MB (261808 pages) 1007 991 975 959 943 927 911 895 879 863 8= 47 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 = 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255= 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) backtrace #0 doadump () at pcpu.h:165 #1 0xc065ff79 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:4= 02 #2 0xc0660240 in panic (fmt=3D0xc08b378a "%s") at /usr/src/sys/kern/kern_shutdown.c:558 #3 0xc087a09e in trap_fatal (frame=3D0xe9606b74, eva=3D0) at /usr/src/sys/i386/i386/trap.c:836 #4 0xc087979a in trap (frame=3D {tf_fs =3D 8, tf_es =3D 40, tf_ds =3D 40, tf_edi =3D -987783168, tf_e= si =3D 0, tf_ebp =3D -379556932, tf_isp =3D -379556960, tf_ebx =3D -9854330= 88, tf_edx =3D -985433088, tf_ecx =3D -985433088, tf_eax =3D 4, tf_trapno = =3D 12, tf_err =3D 0, tf_eip =3D -1066923580, tf_cs =3D 32, tf_eflags =3D 6= 5539, tf_esp =3D 0, tf_ss =3D -987818712}) at /usr/src/sys/i386/i386/trap.c:269 #5 0xc08689aa in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc06809c4 in turnstile_broadcast (ts=3D0x0) at /usr/src/sys/kern/subr_turnstile.c:717 #7 0xc0657587 in _mtx_unlock_sleep (m=3D0xc51f1528, opts=3D0, file=3D0x0, = line=3D0) at /usr/src/sys/kern/kern_mutex.c:676 #8 0xc07fef0c in vnode_pager_lock (first_object=3D0xc5dcfa50) at /usr/src/sys/vm/vnode_pager.c:1198 #9 0xc07eb572 in vm_fault (map=3D0xc591da8c, vaddr=3D134918144,=20 fault_type=3D2 '\002', fault_flags=3D8) at /usr/src/sys/vm/vm_fault.c:2= 99 #10 0xc0879cff in trap_pfault (frame=3D0xe9606d38, usermode=3D1, eva=3D1349= 21272) at /usr/src/sys/i386/i386/trap.c:721 #11 0xc087988f in trap (frame=3D {tf_fs =3D 59, tf_es =3D 59, tf_ds =3D 59, tf_edi =3D 134921272, tf_e= si =3D 672508804, tf_ebp =3D -1077961820, tf_isp =3D -379556508, tf_ebx =3D= 671912920, tf_edx =3D 0, tf_ecx =3D 134534302, tf_eax =3D 671941632, tf_tr= apno =3D 12, tf_err =3D 7, tf_eip =3D 671814244, tf_cs =3D 51, tf_eflags = =3D 66054, tf_esp =3D -1077961844, tf_ss =3D 59}) at /usr/src/sys/i386/i386/trap.c:333 #12 0xc08689aa in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #13 0x280b1264 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) quit --=-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) iQEVAwUBRXdcB/AIdTFWAbdTAQKkswf7Bdp/+qsRQm2UhW3hl7vwidwl+GZv25qO pIcd3+GpwEmMZhfC4h4+CKJyTGfx+qkW8Or3maCqU2mrY5pkOXdxZXEo92w4v15R ae1z3hTvrCAauwQB+ibobdUERYqO3Pp1byY7MJwwVmQgdBJypNoupGXlp3mlvq6o j+7yTieGP9WcsuYqbZGYhyO7+URs+bX3Uy6qL0DbAkjtVVc8u69igJzF+OpD67gc o98uvHkK9+4FLOXPmP03bk3/dO15JM2CN9yJDzOLKj80Vb35Our46aLCgzKzO0oa GCyPd/ihPjMO2UMwYs0cl9F0u/r+m3h3wiSyRQQHP+ugrmdJL97skg== =GZ1D -----END PGP SIGNATURE----- --=-=-=--