From owner-freebsd-stable Sat Apr 13 1:26:53 2002 Delivered-To: freebsd-stable@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id D8FB337B419 for ; Sat, 13 Apr 2002 01:26:31 -0700 (PDT) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id BE8A281488; Sat, 13 Apr 2002 13:03:11 +0930 (CST) Date: Sat, 13 Apr 2002 13:03:11 +0930 From: Greg 'groggy' Lehey To: David =?iso-8859-1?Q?Sieb=F6rger?= Cc: freebsd-stable@freebsd.org Subject: Re: Understanding a crash dump Message-ID: <20020413130311.J47408@wantadilla.lemis.com> References: <20020412224856.A20583@rucus.ru.ac.za> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20020412224856.A20583@rucus.ru.ac.za> User-Agent: Mutt/1.3.23i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Friday, 12 April 2002 at 22:48:56 +0200, David Siebörger wrote: > > One of our FreeBSD machines paniced this afternoon. It saved a crash > dump, but I haven't been able to learn very much from the dump. > > I've attached a backtrace from gdb. Apparently the second panic > occured while trying to sync the disks after the first panic, but the > trail goes cold at #16 and I can't see what caused the original > panic. Yes, it looks as if you're in trouble here. > Can anyone shed any light on this? Is there anything useful still to > be gleaned from this dump, or is there anything that I should do to > get better information in future? Possibly you could get something more out of it, but it would be heavy work. I have some macros in /usr/src/sys/modules/vinum which are really intended for debugging Vinum (thus the location), but which might help here. See http://www.vinumvm.org/vinum/how-to-debug.html for a general description of how to use them, though you won't need the Vinum-specific parts here. In particular, though, you should find a ps command which may show you what was running at the time. I say "may", because you appear to have memory corruption here. > #0 dumpsys () at /usr/build/src/sys/kern/kern_shutdown.c:487 > 487 if (dumping++) { > (kgdb) where > #0 dumpsys () at /usr/build/src/sys/kern/kern_shutdown.c:487 > #1 0xc0169ca3 in boot (howto=260) at /usr/build/src/sys/kern/kern_shutdown.c:316 > #2 0xc016a0e5 in panic (fmt=0xc028bd6c "%s") > at /usr/build/src/sys/kern/kern_shutdown.c:595 > #3 0xc0245eff in trap_fatal (frame=0xc0292d24, eva=48) > at /usr/build/src/sys/i386/i386/trap.c:966 > #4 0xc0245bad in trap_pfault (frame=0xc0292d24, usermode=0, eva=48) > at /usr/build/src/sys/i386/i386/trap.c:859 > #5 0xc0245753 in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, > tf_edi = -1070708352, tf_esi = 0, tf_ebp = -1071043220, tf_isp = -1071043248, > tf_ebx = -1070961092, tf_edx = 6832192, tf_ecx = 31, tf_eax = 0, tf_trapno = 12, > tf_err = 0, tf_eip = -1071861340, tf_cs = 8, tf_eflags = 66050, tf_esp = 0, > tf_ss = 0}) at /usr/build/src/sys/i386/i386/trap.c:458 > #6 0xc01cb1a4 in acquire_lock (lk=0xc02a6e3c) > at /usr/build/src/sys/ufs/ffs/ffs_softdep.c:266 This is the second trap. It might give you some idea about what caused the first trap. > #7 0xc01cf7e2 in softdep_fsync_mountdev (vp=0xd91c4b40) > at /usr/build/src/sys/ufs/ffs/ffs_softdep.c:4024 > #8 0xc01d3b7e in ffs_fsync (ap=0xc0292de0) > at /usr/build/src/sys/ufs/ffs/ffs_vnops.c:134 > #9 0xc01d280b in ffs_sync (mp=0xc1f90200, waitfor=2, cred=0xc16da900, p=0xc02e4980) > at vnode_if.h:558 > #10 0xc0199407 in sync (p=0xc02e4980, uap=0x0) > at /usr/build/src/sys/kern/vfs_syscalls.c:554 > #11 0xc0169a66 in boot (howto=256) at /usr/build/src/sys/kern/kern_shutdown.c:235 > #12 0xc016a0e5 in panic (fmt=0xc028bd6c "%s") > at /usr/build/src/sys/kern/kern_shutdown.c:595 > #13 0xc0245eff in trap_fatal (frame=0xc0292ef8, eva=0) > at /usr/build/src/sys/i386/i386/trap.c:966 > #14 0xc0245bad in trap_pfault (frame=0xc0292ef8, usermode=0, eva=0) > ---Type to continue, or q to quit--- > at /usr/build/src/sys/i386/i386/trap.c:859 > #15 0xc0245753 in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = 0, > tf_esi = -1040807936, tf_ebp = -1071042748, tf_isp = -1071042780, > tf_ebx = -1036232128, tf_edx = -6, tf_ecx = 10, tf_eax = -6, tf_trapno = 12, > tf_err = 0, tf_eip = 0, tf_cs = 8, tf_eflags = 66055, tf_esp = -1040807936, > tf_ss = 40}) at /usr/build/src/sys/i386/i386/trap.c:458 > #16 0x0 in ?? () Somehow you have ended up trying to execute code at address 0. This smells of a smashed stack. I don't think that it would be an indirect function call, since otherwise I'd expect the backtrace to continue. You could find out what the current process is (ps will show it) and use the btp macro to show a backtrace which may show more. Usage is 'btp pid', where pid is the numeric PID of the process. Greg -- See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message