From owner-freebsd-hackers@freebsd.org Fri Oct 2 06:27:21 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98992A0D9B8 for ; Fri, 2 Oct 2015 06:27:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 9D0F01C9E; Fri, 2 Oct 2015 06:27:20 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA04659; Fri, 02 Oct 2015 09:27:19 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1ZhtoM-000PUk-OK; Fri, 02 Oct 2015 09:27:18 +0300 Subject: Re: How to get anything useful out of kgdb? To: Ryan Stone References: <554E41EE.2010202@ignoranthack.me> <2063489.pgabuk9nPJ@ralph.baldwin.cx> <55561803.9050102@ignoranthack.me> <19618854.y3EeXVtCGX@ralph.baldwin.cx> <55561D9A.30309@ignoranthack.me> <555627EC.2020007@ignoranthack.me> Cc: "freebsd-hackers@freebsd.org" , John Baldwin From: Andriy Gapon X-Enigmail-Draft-Status: N1110 Message-ID: <560E238F.9050609@FreeBSD.org> Date: Fri, 2 Oct 2015 09:26:23 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2015 06:27:21 -0000 On 15/05/2015 20:57, Ryan Stone wrote: > *Sigh*, kgdb isn't unwinding the trap frame properly. You can try this to > figure out where it was running: I wonder, what is a reason for this? Can that be fixed in kgdb itself? It seems that usually kgdb handles trap frames just fine, but not always? > That gives you the top of the callstack at the time that the core was > taken. To get the rest of it, try: > > define trace_stack > set $frame_ptr=$arg0 > set $iters=0 > while $frame_ptr != 0 && $iters < $arg1 > set $ret_addr=((char*)$frame_ptr) + sizeof(void*) > printf "frameptr=%p, ret_addr=%p\n", (void*)$frame_ptr, *(void**)$ret_addr > printf " " > info line **(void***)$ret_addr > set $frame_ptr=*(void**)$frame_ptr > set $iters=$iters+1 > end > end > > trace_stack frame->tf_rbp 20 Thank you for this script. Here is an example from my practice. (kgdb) bt #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:291 #1 0xffffffff8063453f in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:359 #2 0xffffffff80634ba4 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:635 #3 0xffffffff806348a3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:568 #4 0xffffffff8041bba7 in db_panic (addr=, have_addr=false, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:473 #5 0xffffffff8041b67b in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:440 #6 0xffffffff8041b524 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493 #7 0xffffffff8041de0b in db_trap (type=, code=0) at /usr/src/sys/ddb/db_main.c:251 #8 0xffffffff80669de8 in kdb_trap (type=19, code=0, tf=0xffffffff80f976d0) at /usr/src/sys/kern/subr_kdb.c:653 #9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at /usr/src/sys/amd64/amd64/trap.c:381 #10 0xffffffff80809623 in nmi_calltrap () at /usr/src/sys/libkern/explicit_bzero.c:28 #11 0xffffffff80619e1f in __mtx_assert (c=, what=, file=, line=) at /usr/src/sys/kern/kern_mutex.c:842 Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) fr 9 #9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at /usr/src/sys/amd64/amd64/trap.c:381 381 kdb_trap(type, 0, frame); (kgdb) trace_stack frame->tf_rbp 20 frameptr=0xfffffe02b8356e90, ret_addr=0xffffffff807fef86 Line 833 of "/usr/src/sys/vm/vm_reserv.c" starts at address 0xffffffff807fef86 and ends at 0xffffffff807fef90 . frameptr=0xfffffe02b8356eb0, ret_addr=0xffffffff807f2b96 Line 2432 of "/usr/src/sys/vm/vm_page.c" starts at address 0xffffffff807f2b96 and ends at 0xffffffff807f2b9c . frameptr=0xfffffe02b8356ed0, ret_addr=0xffffffff807f2e4d Line 963 of "/usr/src/sys/vm/vm_page.c" starts at address 0xffffffff807f2e4d and ends at 0xffffffff807f2e50 . frameptr=0xfffffe02b8356ee0, ret_addr=0xffffffff821c28e2 Line 268 of "/usr/src/sys/modules/drm2/drm2/../../../dev/drm2/ttm/ttm_bo_vm.c" starts at address 0xffffffff821c28e2 and ends at 0xffffffff821c28ee . frameptr=0xfffffe02b8356f50, ret_addr=0xffffffff807d4fd3 Line 321 of "/usr/src/sys/vm/device_pager.c" starts at address 0xffffffff807d4fce and ends at 0xffffffff807d4fdb . frameptr=0xfffffe02b8356fa0, ret_addr=0xffffffff807f9d67 Line 291 of "/usr/src/sys/vm/vm_pager.c" starts at address 0xffffffff807f9d58 and ends at 0xffffffff807f9d6a . frameptr=0xfffffe02b8356fd0, ret_addr=0xffffffff807e0d84 Line 675 of "/usr/src/sys/vm/vm_fault.c" starts at address 0xffffffff807e0d84 and ends at 0xffffffff807e0d8d . frameptr=0xfffffe02b83578f0, ret_addr=0xffffffff807e05ee Line 277 of "/usr/src/sys/vm/vm_fault.c" starts at address 0xffffffff807e05d9 and ends at 0xffffffff807e05f1 . frameptr=0xfffffe02b8357930, ret_addr=0xffffffff80821342 Line 735 of "/usr/src/sys/amd64/amd64/trap.c" starts at address 0xffffffff80821342 and ends at 0xffffffff80821346 . frameptr=0xfffffe02b83579c0, ret_addr=0xffffffff80820bda Line 326 of "/usr/src/sys/amd64/amd64/trap.c" starts at address 0xffffffff80820bc6 and ends at 0xffffffff80820bdf . frameptr=0xfffffe02b8357bd0, ret_addr=0xffffffff8082154a Line 629 of "/usr/src/sys/amd64/amd64/trap.c" starts at address 0xffffffff8082154a and ends at 0xffffffff80821560 . frameptr=0xfffffe02b8357bf0, ret_addr=0xffffffff808091e3 Line 28 of "/usr/src/sys/libkern/explicit_bzero.c" starts at address 0xffffffff806e74dd and ends at 0xffffffff8088a2d0 <__explicit_bzero_hook>. frameptr=0x7fffffffe8f0Cannot access memory at address 0x7fffffffe8f8 Output of trace_stack looks perfectly sane for me up to the next trap frame. -- Andriy Gapon