From owner-freebsd-stable@FreeBSD.ORG Mon Feb 4 23:00:44 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D51B16A51A for ; Mon, 4 Feb 2008 23:00:44 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from speedfactory.net (mail.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id 37A4213C45D for ; Mon, 4 Feb 2008 23:00:44 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8s) with ESMTP id 230850012-1834499 for multiple; Mon, 04 Feb 2008 18:00:44 -0500 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m14N0STb072283; Mon, 4 Feb 2008 18:00:39 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-stable@freebsd.org, Stephen.Clark@seclark.us Date: Mon, 4 Feb 2008 14:59:51 -0500 User-Agent: KMail/1.9.7 References: <47A72CDE.20101@seclark.us> <47A73F6E.8070309@seclark.us> In-Reply-To: <47A73F6E.8070309@seclark.us> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200802041459.51872.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Mon, 04 Feb 2008 18:00:39 -0500 (EST) X-Virus-Scanned: ClamAV 0.91.2/5686/Mon Feb 4 16:13:58 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.1 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00, DATE_IN_PAST_03_06 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Subject: Re: debugging 6.1 crash X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 23:00:44 -0000 On Monday 04 February 2008 11:38:06 am Stephen Clark wrote: > Stephen Clark wrote: > > Hello List, > > > > I am trying to debug a 6.1 panic. When I run kgdb kernel.debug > > /var/crash/vmcore.7 all I get is: > > > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > ... > > > > the info file shows: > > Dump header from device /dev/ad0s1b > > Architecture: i386 > > Architecture Version: 2 > > Dump Length: 116981760B (111 MB) > > Blocksize: 512 > > Dumptime: Mon Feb 4 04:13:09 2008 > > Hostname: G301482.netws.com > > Magic: FreeBSD Kernel Dump > > Version String: FreeBSD 6.1-STABLE #25: Wed Nov 14 10:30:01 EST 2007 > > root@J301002.nwv01.com:/mnt/src/sys/i386/compile/WOLFPAC6SMP > > Panic String: page fault > > Dump Parity: 1156397610 > > Bounds: 7 > > Dump Status: good > > > > > > Does my kernel.debug have to match exactly the crash file kernel. I > > have made the following change > > to my kernel that the kernel.debug is based on. > > --- route.h.orig Tue Apr 4 22:07:23 2006 > > +++ route.h Mon Dec 17 13:11:44 2007 > > @@ -289,6 +289,7 @@ > > #define RT_LOCK_INIT(_rt) \ > > mtx_init(&(_rt)->rt_mtx, "rtentry", NULL, MTX_DEF | MTX_DUPOK) > > #define RT_LOCK(_rt) mtx_lock(&(_rt)->rt_mtx) > > +#define RT_TRYLOCK(_rt) mtx_trylock(&(_rt)->rt_mtx) > > #define RT_UNLOCK(_rt) mtx_unlock(&(_rt)->rt_mtx) > > #define RT_LOCK_DESTROY(_rt) mtx_destroy(&(_rt)->rt_mtx) > > #define RT_LOCK_ASSERT(_rt) mtx_assert(&(_rt)->rt_mtx, > > MA_OWNED) > > --- route.c.orig Tue Oct 30 19:07:54 2007 > > +++ route.c Mon Dec 17 15:13:20 2007 > > @@ -996,6 +996,7 @@ > > struct radix_node_head *rnh = rt_tables[dst->sa_family]; > > int dlen = SA_SIZE(dst), glen = SA_SIZE(gate); > > > > +again: > > RT_LOCK_ASSERT(rt); > > > > /* > > @@ -1029,7 +1030,15 @@ > > RT_REMREF(rt); > > return (EADDRINUSE); /* failure */ > > } > > - RT_LOCK(rt); > > + /* > > + * Try to reacquire the lock on rt, and if it fails, > > + * clean state and restart from scratch. > > + */ > > + if (!RT_TRYLOCK(rt)) { > > + RTFREE_LOCKED(gwrt); > > + RT_LOCK(rt); > > + goto again; > > + } > > /* > > * If there is already a gwroute, then drop it. If we > > * are asked to replace route with itself, then do > > > > Thanks, > > Steve > > > Well I recompiled the kernel without the above changes and I am now kgdb > comes up. It looks like > the panic instruction pointer is in a loadable kernel module. Is there > some way to have kgdb look at the kernel > module? > Below is what i get now: > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd". > > Unread portion of the kernel message buffer: > > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x4 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc0b4536c > stack pointer = 0x28:0xc7516a30 > frame pointer = 0x28:0xc7516a48 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 13 (swi1: net) > trap number = 12 > panic: page fault > cpuid = 0 > Uptime: 8h31m18s > Dumping 111 MB (2 chunks) > chunk 0: 1MB (159 pages) ... ok > chunk 1: 111MB (28400 pages) 95 79 63 47 31 15 > > #0 doadump () at pcpu.h:165 > 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc06492b2 in boot (howto=260) at ../../../kern/kern_shutdown.c:409 > #2 0xc06495d9 in panic (fmt=0xc0910386 "%s") at > ../../../kern/kern_shutdown.c:565 > #3 0xc082d99c in trap_fatal (frame=0xc75169f0, eva=4) at > ../../../i386/i386/trap.c:837 > #4 0xc082d6db in trap_pfault (frame=0xc75169f0, usermode=0, eva=4) > at ../../../i386/i386/trap.c:745 > #5 0xc082d335 in trap (frame= > {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -950965440, tf_esi = > -1026657792, tf_ebp = -950965688, tf_isp = -950965732, tf_ebx = > -1045086208, tf_edx = -1047438316, tf_ecx = 0, tf_eax = 0, tf_trapno = > 12, tf_err = 0, tf_eip = -1061923988, tf_cs = 32, tf_eflags = 590406, > tf_esp = -929974260, tf_ss = 0}) at ../../../i386/i386/trap.c:435 > #6 0xc08198fa in calltrap () at ../../../i386/i386/exception.s:139 > #7 0xc0b4536c in ?? () > Cannot access memory at address 0xc891b80c > (kgdb) list *0xc0b4536c > No source file for address 0xc0b4536c. > (kgdb) > > looking at the loadable kernel modules on the system I get: > sudo kldstat > Id Refs Address Size Name > 1 13 0xc0400000 72862c kernel > 2 1 0xc0b29000 2340 accf_http.ko > 3 1 0xc0b2c000 3b180 ipf.ko > 4 1 0xc0b68000 5c2f8 acpi.ko > 5 1 0xc1dcc000 3000 ng_iface.ko > 6 1 0xc1dcf000 6000 ng_ppp.ko > 7 1 0xc1dd6000 4000 ng_bpf.ko > 8 1 0xc1ddd000 4000 ng_vjc.ko > > Which makes think the panic instruction pointer is in ipf.ko? > This is ipf 4.1.26 compiled out of the kernel tree. > > Steve You can use asf(8) with the -c option to build a .asf file you can source into kgdb to load symbols for the kernel modules to give you a decent backtrace. -- John Baldwin