From owner-freebsd-bugs@FreeBSD.ORG Wed Feb 29 01:20:12 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0F8D106564A for ; Wed, 29 Feb 2012 01:20:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B8E378FC0A for ; Wed, 29 Feb 2012 01:20:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q1T1KBAP065503 for ; Wed, 29 Feb 2012 01:20:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q1T1KBlt065502; Wed, 29 Feb 2012 01:20:11 GMT (envelope-from gnats) Resent-Date: Wed, 29 Feb 2012 01:20:11 GMT Resent-Message-Id: <201202290120.q1T1KBlt065502@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Marc Abramowitz Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E91331065679 for ; Wed, 29 Feb 2012 01:14:33 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id D508B8FC17 for ; Wed, 29 Feb 2012 01:14:33 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q1T1EX2K009934 for ; Wed, 29 Feb 2012 01:14:33 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q1T1EXZ2009933; Wed, 29 Feb 2012 01:14:33 GMT (envelope-from nobody) Message-Id: <201202290114.q1T1EXZ2009933@red.freebsd.org> Date: Wed, 29 Feb 2012 01:14:33 GMT From: Marc Abramowitz To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/165541: Kernel panic when dtracing userland X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Feb 2012 01:20:12 -0000 >Number: 165541 >Category: kern >Synopsis: Kernel panic when dtracing userland >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Feb 29 01:20:11 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Marc Abramowitz >Release: 9.0 >Organization: >Environment: FreeBSD freebsd9-0.localdomain 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Mon Feb 27 00:16:24 PST 2012 root@freebsd9-0.localdomain:/usr/obj/usr/src/sys/DTRACE amd64 >Description: When I dtrace even a trivial program in certain ways, the kernel panics when the traced process exits. I'm using FreeBSD 9.0 on amd64 (I also later reproduced the problem with i386 as well) in VMware Fusion 4.1.1 on OS X 10.6.8 and trying to DTrace userland programs. I recompiled my kernel and world, following the instructions at: http://wiki.freebsd.org/DTrace http://wiki.freebsd.org/DTrace/userland The test.c pid provider example worked fine for me: $ sudo dtrace -s pid.d -c ./test dtrace: script 'pid.d' matched 2 probes dtrace: buffer size lowered to 1m CPU ID FUNCTION:NAME 0 43030 main:entry 0 43031 sleep:entry 0 43031 sleep:entry 0 43031 sleep:entry As does a simple probe of test.c specified with the -n option: [marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c ./test dtrace: description 'pid$target:test:main:entry' matched 1 probe dtrace: buffer size lowered to 1m CPU ID FUNCTION:NAME 0 43030 main:entry When I start trying to dtrace other programs, things don't go so well. DTracing very simple programs causes a kernel panic when the process exits. For example: [marca at freebsd9-0 ~]$ sudo kldload dtraceall [marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:cat:main:entry' -c '/bin/cat hello_world.txt' (kernel panic!) According to the core.txt file, it was a "Fatal trap 10: trace trap while in kernel mode" and here's the KDB backtrace: KDB: stack backtrace: #0 0xffffffff8089025e at kdb_backtrace+0x5e #1 0xffffffff80858ce7 at panic+0x187 #2 0xffffffff80b4bf20 at trap_fatal+0x290 #3 0xffffffff80b4c540 at trap+0x180 #4 0xffffffff80b36963 at calltrap+0x8 #5 0xffffffff8162583d at dtrace_assfail+0x2d #6 0xffffffff8188aa2e at fasttrap_provider_free+0x1de #7 0xffffffff8188ad13 at fasttrap_pid_cleanup_cb+0x1c3 #8 0xffffffff8086dfa1 at softclock+0x3a1 #9 0xffffffff8082d724 at intr_event_execute_handlers+0x104 #10 0xffffffff8082eee4 at ithread_loop+0xa4 #11 0xffffffff8082a34f at fork_exit+0x11f #12 0xffffffff80b36e8e at fork_trampoline+0xe >How-To-Repeat: [marca at freebsd9-0 ~]$ sudo kldload dtraceall [marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:cat:main:entry' -c '/bin/cat hello_world.txt' - or - [marca at freebsd9-0 ~]$ sudo kldload dtraceall [marca at freebsd9-0 ~]$ cat -n test.c 1 #include 2 3 int main() 4 { 5 sleep(15); 6 7 FILE *fp = fopen("hello.txt", "w"); 8 fprintf(fp, "Here I am at %s:%d.\n", __FILE__, __LINE__); 9 fclose(fp); 10 } [marca at freebsd9-0 ~]$ gcc test.c -o test [marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c ./test dtrace: description 'pid$target:test:main:entry' matched 1 probe dtrace: buffer size lowered to 1m CPU ID FUNCTION:NAME 0 43030 main:entry (Kernel panic! After reboot....) [marca at freebsd9-0 ~]$ cat hello.txt Here I am at test.c:8. Interestingly, the crash doesn't occur until after the sleep and the fprintf call, so it looks the kernel panic happens as a result of the traced process _exiting_... I reproduced this on both the amd64 and i386 architectures. >Fix: >Release-Note: >Audit-Trail: >Unformatted: