Date: Mon, 4 Feb 2008 14:59:51 -0500 From: John Baldwin <jhb@freebsd.org> To: freebsd-stable@freebsd.org, Stephen.Clark@seclark.us Subject: Re: debugging 6.1 crash Message-ID: <200802041459.51872.jhb@freebsd.org> In-Reply-To: <47A73F6E.8070309@seclark.us> References: <47A72CDE.20101@seclark.us> <47A73F6E.8070309@seclark.us>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday 04 February 2008 11:38:06 am Stephen Clark wrote:
> Stephen Clark wrote:
> > Hello List,
> >
> > I am trying to debug a 6.1 panic. When I run kgdb kernel.debug
> > /var/crash/vmcore.7 all I get is:
> >
> > kgdb: kvm_read: invalid address (0x24)
> > kgdb: kvm_read: invalid address (0x24)
> > kgdb: kvm_read: invalid address (0x24)
> > kgdb: kvm_read: invalid address (0x24)
> > kgdb: kvm_read: invalid address (0x24)
> > kgdb: kvm_read: invalid address (0x24)
> > kgdb: kvm_read: invalid address (0x24)
> > ...
> >
> > the info file shows:
> > Dump header from device /dev/ad0s1b
> > Architecture: i386
> > Architecture Version: 2
> > Dump Length: 116981760B (111 MB)
> > Blocksize: 512
> > Dumptime: Mon Feb 4 04:13:09 2008
> > Hostname: G301482.netws.com
> > Magic: FreeBSD Kernel Dump
> > Version String: FreeBSD 6.1-STABLE #25: Wed Nov 14 10:30:01 EST 2007
> > root@J301002.nwv01.com:/mnt/src/sys/i386/compile/WOLFPAC6SMP
> > Panic String: page fault
> > Dump Parity: 1156397610
> > Bounds: 7
> > Dump Status: good
> >
> >
> > Does my kernel.debug have to match exactly the crash file kernel. I
> > have made the following change
> > to my kernel that the kernel.debug is based on.
> > --- route.h.orig Tue Apr 4 22:07:23 2006
> > +++ route.h Mon Dec 17 13:11:44 2007
> > @@ -289,6 +289,7 @@
> > #define RT_LOCK_INIT(_rt) \
> > mtx_init(&(_rt)->rt_mtx, "rtentry", NULL, MTX_DEF | MTX_DUPOK)
> > #define RT_LOCK(_rt) mtx_lock(&(_rt)->rt_mtx)
> > +#define RT_TRYLOCK(_rt) mtx_trylock(&(_rt)->rt_mtx)
> > #define RT_UNLOCK(_rt) mtx_unlock(&(_rt)->rt_mtx)
> > #define RT_LOCK_DESTROY(_rt) mtx_destroy(&(_rt)->rt_mtx)
> > #define RT_LOCK_ASSERT(_rt) mtx_assert(&(_rt)->rt_mtx,
> > MA_OWNED)
> > --- route.c.orig Tue Oct 30 19:07:54 2007
> > +++ route.c Mon Dec 17 15:13:20 2007
> > @@ -996,6 +996,7 @@
> > struct radix_node_head *rnh = rt_tables[dst->sa_family];
> > int dlen = SA_SIZE(dst), glen = SA_SIZE(gate);
> >
> > +again:
> > RT_LOCK_ASSERT(rt);
> >
> > /*
> > @@ -1029,7 +1030,15 @@
> > RT_REMREF(rt);
> > return (EADDRINUSE); /* failure */
> > }
> > - RT_LOCK(rt);
> > + /*
> > + * Try to reacquire the lock on rt, and if it fails,
> > + * clean state and restart from scratch.
> > + */
> > + if (!RT_TRYLOCK(rt)) {
> > + RTFREE_LOCKED(gwrt);
> > + RT_LOCK(rt);
> > + goto again;
> > + }
> > /*
> > * If there is already a gwroute, then drop it. If we
> > * are asked to replace route with itself, then do
> >
> > Thanks,
> > Steve
> >
> Well I recompiled the kernel without the above changes and I am now kgdb
> comes up. It looks like
> the panic instruction pointer is in a loadable kernel module. Is there
> some way to have kgdb look at the kernel
> module?
> Below is what i get now:
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "i386-marcel-freebsd".
>
> Unread portion of the kernel message buffer:
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0x4
> fault code = supervisor read, page not present
> instruction pointer = 0x20:0xc0b4536c
> stack pointer = 0x28:0xc7516a30
> frame pointer = 0x28:0xc7516a48
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, def32 1, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 13 (swi1: net)
> trap number = 12
> panic: page fault
> cpuid = 0
> Uptime: 8h31m18s
> Dumping 111 MB (2 chunks)
> chunk 0: 1MB (159 pages) ... ok
> chunk 1: 111MB (28400 pages) 95 79 63 47 31 15
>
> #0 doadump () at pcpu.h:165
> 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
> (kgdb) bt
> #0 doadump () at pcpu.h:165
> #1 0xc06492b2 in boot (howto=260) at ../../../kern/kern_shutdown.c:409
> #2 0xc06495d9 in panic (fmt=0xc0910386 "%s") at
> ../../../kern/kern_shutdown.c:565
> #3 0xc082d99c in trap_fatal (frame=0xc75169f0, eva=4) at
> ../../../i386/i386/trap.c:837
> #4 0xc082d6db in trap_pfault (frame=0xc75169f0, usermode=0, eva=4)
> at ../../../i386/i386/trap.c:745
> #5 0xc082d335 in trap (frame=
> {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -950965440, tf_esi =
> -1026657792, tf_ebp = -950965688, tf_isp = -950965732, tf_ebx =
> -1045086208, tf_edx = -1047438316, tf_ecx = 0, tf_eax = 0, tf_trapno =
> 12, tf_err = 0, tf_eip = -1061923988, tf_cs = 32, tf_eflags = 590406,
> tf_esp = -929974260, tf_ss = 0}) at ../../../i386/i386/trap.c:435
> #6 0xc08198fa in calltrap () at ../../../i386/i386/exception.s:139
> #7 0xc0b4536c in ?? ()
> Cannot access memory at address 0xc891b80c
> (kgdb) list *0xc0b4536c
> No source file for address 0xc0b4536c.
> (kgdb)
>
> looking at the loadable kernel modules on the system I get:
> sudo kldstat
> Id Refs Address Size Name
> 1 13 0xc0400000 72862c kernel
> 2 1 0xc0b29000 2340 accf_http.ko
> 3 1 0xc0b2c000 3b180 ipf.ko
> 4 1 0xc0b68000 5c2f8 acpi.ko
> 5 1 0xc1dcc000 3000 ng_iface.ko
> 6 1 0xc1dcf000 6000 ng_ppp.ko
> 7 1 0xc1dd6000 4000 ng_bpf.ko
> 8 1 0xc1ddd000 4000 ng_vjc.ko
>
> Which makes think the panic instruction pointer is in ipf.ko?
> This is ipf 4.1.26 compiled out of the kernel tree.
>
> Steve
You can use asf(8) with the -c option to build a .asf file you can source into
kgdb to load symbols for the kernel modules to give you a decent backtrace.
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200802041459.51872.jhb>