Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 29 Feb 2012 01:14:33 GMT
From:      Marc Abramowitz <marc@marc-abramowitz.com>
To:        freebsd-gnats-submit@FreeBSD.org
Subject:   kern/165541: Kernel panic when dtracing userland
Message-ID:  <201202290114.q1T1EXZ2009933@red.freebsd.org>
Resent-Message-ID: <201202290120.q1T1KBlt065502@freefall.freebsd.org>

next in thread | raw e-mail | index | archive | help

>Number:         165541
>Category:       kern
>Synopsis:       Kernel panic when dtracing userland
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 29 01:20:11 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Marc Abramowitz
>Release:        9.0
>Organization:
>Environment:
FreeBSD freebsd9-0.localdomain 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Mon Feb 27 00:16:24 PST 2012    root@freebsd9-0.localdomain:/usr/obj/usr/src/sys/DTRACE amd64
>Description:
When I dtrace even a trivial program in certain ways, the kernel panics when the traced process exits.

I'm using FreeBSD 9.0 on amd64 (I also later reproduced the problem with i386 as well) in VMware Fusion 4.1.1 on OS X 10.6.8 and trying to DTrace userland programs.

I recompiled my kernel and world, following the instructions at:

http://wiki.freebsd.org/DTrace
http://wiki.freebsd.org/DTrace/userland

The test.c pid provider example worked fine for me:

$ sudo dtrace -s pid.d -c ./test
dtrace: script 'pid.d' matched 2 probes
dtrace: buffer size lowered to 1m
CPU     ID                    FUNCTION:NAME
  0  43030                       main:entry
  0  43031                      sleep:entry
  0  43031                      sleep:entry
  0  43031                      sleep:entry

As does a simple probe of test.c specified with the -n option:

[marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c ./test
dtrace: description 'pid$target:test:main:entry' matched 1 probe
dtrace: buffer size lowered to 1m
CPU     ID                    FUNCTION:NAME
  0  43030                       main:entry

When I start trying to dtrace other programs, things don't go so well.

DTracing very simple programs causes a kernel panic when the process exits. For example:

[marca at freebsd9-0 ~]$ sudo kldload dtraceall
[marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:cat:main:entry' -c '/bin/cat hello_world.txt'
(kernel panic!)

According to the core.txt file, it was a "Fatal trap 10: trace trap while
in kernel mode" and here's the KDB backtrace:

KDB: stack backtrace:
#0 0xffffffff8089025e at kdb_backtrace+0x5e
#1 0xffffffff80858ce7 at panic+0x187
#2 0xffffffff80b4bf20 at trap_fatal+0x290
#3 0xffffffff80b4c540 at trap+0x180
#4 0xffffffff80b36963 at calltrap+0x8
#5 0xffffffff8162583d at dtrace_assfail+0x2d
#6 0xffffffff8188aa2e at fasttrap_provider_free+0x1de
#7 0xffffffff8188ad13 at fasttrap_pid_cleanup_cb+0x1c3
#8 0xffffffff8086dfa1 at softclock+0x3a1
#9 0xffffffff8082d724 at intr_event_execute_handlers+0x104
#10 0xffffffff8082eee4 at ithread_loop+0xa4
#11 0xffffffff8082a34f at fork_exit+0x11f
#12 0xffffffff80b36e8e at fork_trampoline+0xe
>How-To-Repeat:
[marca at freebsd9-0 ~]$ sudo kldload dtraceall
[marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:cat:main:entry' -c '/bin/cat hello_world.txt'

- or -

[marca at freebsd9-0 ~]$ sudo kldload dtraceall
[marca at freebsd9-0 ~]$ cat -n test.c
     1 #include <stdio.h>
     2
     3 int main()
     4 {
     5    sleep(15);
     6
     7    FILE *fp = fopen("hello.txt", "w");
     8    fprintf(fp, "Here I am at %s:%d.\n", __FILE__, __LINE__);
     9    fclose(fp);
    10 }
[marca at freebsd9-0 ~]$ gcc test.c -o test
[marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c ./test
dtrace: description 'pid$target:test:main:entry' matched 1 probe
dtrace: buffer size lowered to 1m
CPU     ID                    FUNCTION:NAME
  0  43030                       main:entry
(Kernel panic!  After reboot....)
[marca at freebsd9-0 ~]$ cat hello.txt
Here I am at test.c:8.

Interestingly, the crash doesn't occur until after the sleep and the
fprintf call, so it looks the kernel panic happens as a result of the
traced process _exiting_...

I reproduced this on both the amd64 and i386 architectures.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201202290114.q1T1EXZ2009933>