From owner-freebsd-questions@FreeBSD.ORG Tue Aug 19 12:46:58 2003 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 07D8316A4BF for ; Tue, 19 Aug 2003 12:46:58 -0700 (PDT) Received: from poecilotheria.netmails.net (netmails.net [12.96.164.138]) by mx1.FreeBSD.org (Postfix) with SMTP id 135A043FBD for ; Tue, 19 Aug 2003 12:46:57 -0700 (PDT) (envelope-from subscr@poecilotheria.netmails.net) Received: (qmail 22050 invoked by uid 1012); 19 Aug 2003 19:46:55 -0000 Date: Tue, 19 Aug 2003 14:46:55 -0500 From: Hari Bhaskaran To: Greg 'groggy' Lehey Message-ID: <20030819194655.GA21974@poecilotheria.netmails.net> References: <20030817200141.GA10480@poecilotheria.netmails.net> <20030817235159.GZ15411@wantadilla.lemis.com> <20030819031532.GA17344@poecilotheria.netmails.net> <20030819135226.GA19663@poecilotheria.netmails.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030819135226.GA19663@poecilotheria.netmails.net> User-Agent: Mutt/1.4i cc: FreeBSD Questions Subject: Re: panic on heavy read/writes (5.1 + vinum) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Aug 2003 19:46:58 -0000 Prev trace was not using kernel.debug, I found kernel.debug file that matched the running kernel and here is the trace with it (no errors from gdb now). BTW, I will be moving back to 4.8+vinum tonight ettukaali# gdb -k kernel.debug vmcore.0 GNU gdb 5.2.1 (FreeBSD) Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-undermydesk-freebsd"... panic: free locked buf panic messages: --- Fatal trap 12: page fault while in kernel mode fault virtual address = 0x14 fault code = supervisor write, page not present instruction pointer = 0x8:0xc02d15f9 stack pointer = 0x10:0xd257d858 frame pointer = 0x10:0xd257d88c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 94137 (genattrtab) trap number = 12 panic: page fault syncing disks, buffers remaining... 2220 2220 panic: free locked buf Uptime: 5h12m33s Dumping 256 MB ata0: resetting devices .. done 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 --- Reading symbols from /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko.debug...done. Loaded symbols for /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko.debug #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:238 238 dumping++; add symbol table from file "/usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko.debug" at .text_addr = 0x0 (kgdb) f 1 #1 0xc0303b9a in boot (howto=0x104) at /usr/src/sys/kern/kern_shutdown.c:370 370 doadump(); (kgdb) bt #0 doadump () at /usr/src/sys/kern/kern_shutdown.c:238 #1 0xc0303b9a in boot (howto=0x104) at /usr/src/sys/kern/kern_shutdown.c:370 #2 0xc0303e53 in panic () at /usr/src/sys/kern/kern_shutdown.c:543 #3 0xc2758926 in freerq (rq=0xc2ebc8c0) at /usr/src/sys/dev/vinum/vinuminterrupt.c:252 #4 0xc275882a in complete_rqe (bp=0xc2633024) at /usr/src/sys/dev/vinum/vinuminterrupt.c:230 #5 0xc0346451 in bufdone (bp=0xc2633024) at /usr/src/sys/kern/vfs_bio.c:3086 #6 0xc0346354 in bufdonebio (bp=0x0) at /usr/src/sys/kern/vfs_bio.c:3034 #7 0xc03461b2 in biodone (bp=0xc2633024) at /usr/src/sys/kern/vfs_bio.c:2961 #8 0xc02d159e in g_dev_done (bp2=0xc2634090) at /usr/src/sys/geom/geom_dev.c:391 #9 0xc03461b2 in biodone (bp=0xc2634090) at /usr/src/sys/kern/vfs_bio.c:2961 #10 0xc02d3d42 in g_io_schedule_up (tp=0xc0ecbe40) at /usr/src/sys/geom/geom_io.c:365 #11 0xc02d3f38 in g_up_procbody () at /usr/src/sys/geom/geom_kern.c:91 #12 0xc02f109e in fork_exit (callout=0xc02d3f10 , arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:768 (kgdb) On Tue, Aug 19, 2003 at 08:52:26AM -0500, Hari Bhaskaran wrote: > There are two files in > /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum > One is vinum.ko and the other is vinum.ko.debug (Both are not stripped) > I have tried the gdb dump with vinum.paths pointing to either one > of them and it still gives the error mentioned below ("Attempt to extract..") > > -- > Hari > > On Mon, Aug 18, 2003 at 10:15:32PM -0500, Hari Bhaskaran wrote: > > ok, here is it, trace for kernel panic > > > > NOTE: There seems to a problem in the .gdbinit files > > (I may need your help on that). > > > > panic: free locked buf > > panic messages: > > --- > > Fatal trap 12: page fault while in kernel mode > > fault virtual address = 0x14 > > fault code = supervisor write, page not present > > instruction pointer = 0x8:0xc02d15f9 > > stack pointer = 0x10:0xd257d858 > > frame pointer = 0x10:0xd257d88c > > code segment = base 0x0, limit 0xfffff, type 0x1b > > = DPL 0, pres 1, def32 1, gran 1 > > processor eflags = interrupt enabled, resume, IOPL = 0 > > current process = 94137 (genattrtab) > > trap number = 12 > > panic: page fault > > > > syncing disks, buffers remaining... 2220 2220 panic: free locked buf > > Uptime: 5h12m33s > > Dumping 256 MB > > ata0: resetting devices .. > > done > > 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 > > --- > > > > And here is the output on gdb f & bt commands > > > > (kgdb) f 1 > > #1 0xc0303b9a in boot () > > (kgdb) bt > > #0 0xc030366b in doadump () > > #1 0xc0303b9a in boot () > > #2 0xc0303e53 in panic () > > #3 0xc2758926 in freerq (rq=0xc2ebc8c0) at /usr/src/sys/dev/vinum/vinuminterrupt.c:252 > > #4 0xc275882a in complete_rqe (bp=0xc2633024) at /usr/src/sys/dev/vinum/vinuminterrupt.c: > > 230 > > #5 0xc0346451 in bufdone () > > #6 0xc0346354 in bufdonebio () > > #7 0xc03461b2 in biodone () > > #8 0xc02d159e in g_dev_done () > > #9 0xc03461b2 in biodone () > > #10 0xc02d3d42 in g_io_schedule_up () > > #11 0xc02d3f38 in g_up_procbody () > > #12 0xc02f109e in fork_exit () > > > > Now this trace may be wrong because gdb gave this error when it loaded. > > > > Reading symbols from /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko > > .debug...done. > > Loaded symbols for /usr/obj/usr/src/sys/DEBUG/modules/usr/src/sys/modules/vinum/vinum.ko.d > > ebug > > #0 0xc030366b in doadump () > > .gdbinit:4: Error in sourced command file: > > Attempt to extract a component of a value that is not a structure. > > > > I have all these in separate files (output, gdbinit files etc) in a zip. > > > > BTW, I was doing a make buildworld while the machine crashed. > > And yes, both kernel and vinum.ko have debugging info. > > > > ettukaali# file /boot/kernel/kernel > > /boot/kernel/kernel: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), dynamically linked (uses shared libs), not stripped > > ettukaali# ls -al /boot/kernel/kernel > > -r-xr-xr-x 1 root wheel 5455085 Aug 17 23:07 /boot/kernel/kernel > > ettukaali# file /boot/kernel/vinum.ko > > /boot/kernel/vinum.ko: ELF 32-bit LSB shared object, Intel 80386, version 1 (FreeBSD), not stripped > > ettukaali# ls -al /boot/kernel/vinum.ko > > -r-xr-xr-x 1 root wheel 96588 Aug 18 07:34 /boot/kernel/vinum.ko > > ettukaali# > > > > (Aug 17 23:07 is approx when I made the debug kernel) > > (Aug 18 07:34 is approx when I did make installkernel - although I would > > have expected the last-mod-time on vinum.ko to match that of kernel) > > > > Either-way the make buildworld wouldn't have affected any of these files > > since I did it in the afternoon. > > > > -- > > Hari > > > > > > On Mon, Aug 18, 2003 at 09:21:59AM +0930, Greg 'groggy' Lehey wrote: > > > Follow the steps at > > > http://www.vinumvm.org/vinum/how-to-debug.html#panic. I can't help > > > you if you don't tell me what's wrong. > > > > > _______________________________________________ > > freebsd-questions@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"