From owner-freebsd-hackers@freebsd.org Fri Oct 2 16:17:15 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E82FA0DEBC for ; Fri, 2 Oct 2015 16:17:15 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5ACEF1612; Fri, 2 Oct 2015 16:17:15 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 7A346B946; Fri, 2 Oct 2015 12:17:13 -0400 (EDT) From: John Baldwin To: Andriy Gapon Cc: Ryan Stone , "freebsd-hackers@freebsd.org" Subject: Re: How to get anything useful out of kgdb? Date: Fri, 02 Oct 2015 09:12:58 -0700 Message-ID: <1595419.L0rkNTMkPe@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-PRERELEASE; KDE/4.14.3; amd64; ; ) In-Reply-To: <560E238F.9050609@FreeBSD.org> References: <554E41EE.2010202@ignoranthack.me> <560E238F.9050609@FreeBSD.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 02 Oct 2015 12:17:13 -0400 (EDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 16:17:15 -0000 On Friday, October 02, 2015 09:26:23 AM Andriy Gapon wrote: > On 15/05/2015 20:57, Ryan Stone wrote: > > *Sigh*, kgdb isn't unwinding the trap frame properly. You can try this to > > figure out where it was running: > > I wonder, what is a reason for this? > Can that be fixed in kgdb itself? > It seems that usually kgdb handles trap frames just fine, but not always? It should be fixable. If this doesn't work under newer kgdb let me know and I'll try to fix it. I did fix a few edge cases with special frame handling in the newer kgdb though those mostly had to do with fork_trampoline and possibly Xtimerint (and aside from fork_trampoline I think the fixes were mostly for i386 where different handlers setup trapframes differently) > > That gives you the top of the callstack at the time that the core was > > taken. To get the rest of it, try: > > > > define trace_stack > > set $frame_ptr=$arg0 > > set $iters=0 > > while $frame_ptr != 0 && $iters < $arg1 > > set $ret_addr=((char*)$frame_ptr) + sizeof(void*) > > printf "frameptr=%p, ret_addr=%p\n", (void*)$frame_ptr, *(void**)$ret_addr > > printf " " > > info line **(void***)$ret_addr > > set $frame_ptr=*(void**)$frame_ptr > > set $iters=$iters+1 > > end > > end > > > > trace_stack frame->tf_rbp 20 > > Thank you for this script. > Here is an example from my practice. > > (kgdb) bt > #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:291 > #1 0xffffffff8063453f in kern_reboot (howto=260) at > /usr/src/sys/kern/kern_shutdown.c:359 > #2 0xffffffff80634ba4 in vpanic (fmt=, ap= out>) at /usr/src/sys/kern/kern_shutdown.c:635 > #3 0xffffffff806348a3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:568 > #4 0xffffffff8041bba7 in db_panic (addr=, have_addr=false, > count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:473 > #5 0xffffffff8041b67b in db_command (cmd_table=0x0) at > /usr/src/sys/ddb/db_command.c:440 > #6 0xffffffff8041b524 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493 > #7 0xffffffff8041de0b in db_trap (type=, code=0) at > /usr/src/sys/ddb/db_main.c:251 > #8 0xffffffff80669de8 in kdb_trap (type=19, code=0, tf=0xffffffff80f976d0) at > /usr/src/sys/kern/subr_kdb.c:653 > #9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at > /usr/src/sys/amd64/amd64/trap.c:381 > #10 0xffffffff80809623 in nmi_calltrap () at > /usr/src/sys/libkern/explicit_bzero.c:28 This may be part of the problem. The trapframe unwinder depends on function names to know when it is crossing a trapframe. nmi_calltrap() is not the function at explicit_bzero.c:28. Usually debugging this sort of thing starts by going to frame 11 and comparing its registers with the values in the trapframe. They should match, but sometimes you will find them shifted by one or two, etc. -- John Baldwin