Date: Tue, 19 Aug 2003 14:46:55 -0500 From: Hari Bhaskaran <subscr@spider.netmails.net> To: Greg 'groggy' Lehey <grog@FreeBSD.org> Cc: FreeBSD Questions <questions@FreeBSD.org> Subject: Re: panic on heavy read/writes (5.1 + vinum) Message-ID: <20030819194655.GA21974@poecilotheria.netmails.net> In-Reply-To: <20030819135226.GA19663@poecilotheria.netmails.net> References: <20030817200141.GA10480@poecilotheria.netmails.net> <20030817235159.GZ15411@wantadilla.lemis.com> <20030819031532.GA17344@poecilotheria.netmails.net> <20030819135226.GA19663@poecilotheria.netmails.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Prev trace was not using kernel.debug, I found kernel.debug
file that matched the running kernel and here is the trace
with it (no errors from gdb now).
BTW, I will be moving back to 4.8+vinum tonight
ettukaali# gdb -k kernel.debug vmcore.0
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic: free locked buf
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x14
fault code = supervisor write, page not present
instruction pointer = 0x8:0xc02d15f9
stack pointer = 0x10:0xd257d858
frame pointer = 0x10:0xd257d88c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 94137 (genattrtab)
trap number = 12
panic: page fault
syncing disks, buffers remaining... 2220 2220 panic: free locked buf
Uptime: 5h12m33s
Dumping 256 MB
ata0: resetting devices ..
done
16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
---
Reading symbols from /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko.debug...done.
Loaded symbols for /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko.debug
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:238
238 dumping++;
add symbol table from file "/usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko.debug" at
.text_addr = 0x0
(kgdb) f 1
#1 0xc0303b9a in boot (howto=0x104) at /usr/src/sys/kern/kern_shutdown.c:370
370 doadump();
(kgdb) bt
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:238
#1 0xc0303b9a in boot (howto=0x104) at /usr/src/sys/kern/kern_shutdown.c:370
#2 0xc0303e53 in panic () at /usr/src/sys/kern/kern_shutdown.c:543
#3 0xc2758926 in freerq (rq=0xc2ebc8c0) at /usr/src/sys/dev/vinum/vinuminterrupt.c:252
#4 0xc275882a in complete_rqe (bp=0xc2633024) at /usr/src/sys/dev/vinum/vinuminterrupt.c:230
#5 0xc0346451 in bufdone (bp=0xc2633024) at /usr/src/sys/kern/vfs_bio.c:3086
#6 0xc0346354 in bufdonebio (bp=0x0) at /usr/src/sys/kern/vfs_bio.c:3034
#7 0xc03461b2 in biodone (bp=0xc2633024) at /usr/src/sys/kern/vfs_bio.c:2961
#8 0xc02d159e in g_dev_done (bp2=0xc2634090) at /usr/src/sys/geom/geom_dev.c:391
#9 0xc03461b2 in biodone (bp=0xc2634090) at /usr/src/sys/kern/vfs_bio.c:2961
#10 0xc02d3d42 in g_io_schedule_up (tp=0xc0ecbe40) at /usr/src/sys/geom/geom_io.c:365
#11 0xc02d3f38 in g_up_procbody () at /usr/src/sys/geom/geom_kern.c:91
#12 0xc02f109e in fork_exit (callout=0xc02d3f10 <g_up_procbody>, arg=0x0, frame=0x0)
at /usr/src/sys/kern/kern_fork.c:768
(kgdb)
On Tue, Aug 19, 2003 at 08:52:26AM -0500, Hari Bhaskaran wrote:
> There are two files in
> /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum
> One is vinum.ko and the other is vinum.ko.debug (Both are not stripped)
> I have tried the gdb dump with vinum.paths pointing to either one
> of them and it still gives the error mentioned below ("Attempt to extract..")
>
> --
> Hari
>
> On Mon, Aug 18, 2003 at 10:15:32PM -0500, Hari Bhaskaran wrote:
> > ok, here is it, trace for kernel panic
> >
> > NOTE: There seems to a problem in the .gdbinit files
> > (I may need your help on that).
> >
> > panic: free locked buf
> > panic messages:
> > ---
> > Fatal trap 12: page fault while in kernel mode
> > fault virtual address = 0x14
> > fault code = supervisor write, page not present
> > instruction pointer = 0x8:0xc02d15f9
> > stack pointer = 0x10:0xd257d858
> > frame pointer = 0x10:0xd257d88c
> > code segment = base 0x0, limit 0xfffff, type 0x1b
> > = DPL 0, pres 1, def32 1, gran 1
> > processor eflags = interrupt enabled, resume, IOPL = 0
> > current process = 94137 (genattrtab)
> > trap number = 12
> > panic: page fault
> >
> > syncing disks, buffers remaining... 2220 2220 panic: free locked buf
> > Uptime: 5h12m33s
> > Dumping 256 MB
> > ata0: resetting devices ..
> > done
> > 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
> > ---
> >
> > And here is the output on gdb f & bt commands
> >
> > (kgdb) f 1
> > #1 0xc0303b9a in boot ()
> > (kgdb) bt
> > #0 0xc030366b in doadump ()
> > #1 0xc0303b9a in boot ()
> > #2 0xc0303e53 in panic ()
> > #3 0xc2758926 in freerq (rq=0xc2ebc8c0) at /usr/src/sys/dev/vinum/vinuminterrupt.c:252
> > #4 0xc275882a in complete_rqe (bp=0xc2633024) at /usr/src/sys/dev/vinum/vinuminterrupt.c:
> > 230
> > #5 0xc0346451 in bufdone ()
> > #6 0xc0346354 in bufdonebio ()
> > #7 0xc03461b2 in biodone ()
> > #8 0xc02d159e in g_dev_done ()
> > #9 0xc03461b2 in biodone ()
> > #10 0xc02d3d42 in g_io_schedule_up ()
> > #11 0xc02d3f38 in g_up_procbody ()
> > #12 0xc02f109e in fork_exit ()
> >
> > Now this trace may be wrong because gdb gave this error when it loaded.
> >
> > Reading symbols from /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko
> > .debug...done.
> > Loaded symbols for /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko.d
> > ebug
> > #0 0xc030366b in doadump ()
> > .gdbinit:4: Error in sourced command file:
> > Attempt to extract a component of a value that is not a structure.
> >
> > I have all these in separate files (output, gdbinit files etc) in a zip.
> >
> > BTW, I was doing a make buildworld while the machine crashed.
> > And yes, both kernel and vinum.ko have debugging info.
> >
> > ettukaali# file /boot/kernel/kernel
> > /boot/kernel/kernel: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), dynamically linked (uses shared libs), not stripped
> > ettukaali# ls -al /boot/kernel/kernel
> > -r-xr-xr-x 1 root wheel 5455085 Aug 17 23:07 /boot/kernel/kernel
> > ettukaali# file /boot/kernel/vinum.ko
> > /boot/kernel/vinum.ko: ELF 32-bit LSB shared object, Intel 80386, version 1 (FreeBSD), not stripped
> > ettukaali# ls -al /boot/kernel/vinum.ko
> > -r-xr-xr-x 1 root wheel 96588 Aug 18 07:34 /boot/kernel/vinum.ko
> > ettukaali#
> >
> > (Aug 17 23:07 is approx when I made the debug kernel)
> > (Aug 18 07:34 is approx when I did make installkernel - although I would
> > have expected the last-mod-time on vinum.ko to match that of kernel)
> >
> > Either-way the make buildworld wouldn't have affected any of these files
> > since I did it in the afternoon.
> >
> > --
> > Hari
> >
> >
> > On Mon, Aug 18, 2003 at 09:21:59AM +0930, Greg 'groggy' Lehey wrote:
> > > Follow the steps at
> > > http://www.vinumvm.org/vinum/how-to-debug.html#panic. I can't help
> > > you if you don't tell me what's wrong.
> >
>
> > _______________________________________________
> > freebsd-questions@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"
>
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030819194655.GA21974>