From owner-freebsd-dtrace@FreeBSD.ORG Wed Jun 26 19:47:33 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9C11059E for ; Wed, 26 Jun 2013 19:47:33 +0000 (UTC) (envelope-from as4idov@gmail.com) Received: from mail-qc0-x241.google.com (mail-qc0-x241.google.com [IPv6:2607:f8b0:400d:c01::241]) by mx1.freebsd.org (Postfix) with ESMTP id 61DA118DC for ; Wed, 26 Jun 2013 19:47:33 +0000 (UTC) Received: by mail-qc0-f193.google.com with SMTP id c1so4232154qcz.0 for ; Wed, 26 Jun 2013 12:47:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:x-google-sender-delegation:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=mp8hCU4MT8WS52JgzcvSYCCVbXmYJjk/asP1tFnc28g=; b=vPZ4YWlDMmF0qa4rWDURm7PJaEagC4hiWd4r0VBYm9t+hUt5OYmS+PgYi7wXoe/qHx shKXjm4NsSj4oHFYmv/OsDezDL61fISmNveEfDUX7Cl2A3cWHZrb8l02QT/KHy2bICOA ijDHFOlSKiagC3eEuX2M5+idgu+TSOuiWVWmCIS9RUcm1r03Q3aFXsMntCdvcM3k42Mu uYV0lg1qC9wrEAYPy8h5V0lC4qHc/sUxeb1jVvaJ5k8iOWm2GRaZHQBf1uKsVLsuiE8+ gRFGlDEVATbIEIE56A0CFxtYyVnYmiQlkxfNn07erKeMHqL0mLuGPBaKR1N3kWFdPrQI cljA== MIME-Version: 1.0 X-Received: by 10.229.172.10 with SMTP id j10mr1537002qcz.110.1372276052870; Wed, 26 Jun 2013 12:47:32 -0700 (PDT) Sender: xbro.aka.tj@gmail.com X-Google-Sender-Delegation: xbro.aka.tj@gmail.com Received: by 10.224.4.136 with HTTP; Wed, 26 Jun 2013 12:47:32 -0700 (PDT) Date: Wed, 26 Jun 2013 12:47:32 -0700 X-Google-Sender-Auth: hy1GVtsRwnlDlegDup2LYjv23Uc Message-ID: Subject: DTrace userland support doesn't work on post r219107 FreeBSD 8.x From: Akbar Saidov To: freebsd-dtrace@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2013 19:47:33 -0000 Hi Guys, George Neville-Neil suggested me to email this list describing the issue. I am currently working on backporting DTrace userland support to FreeBSD 8.2-Release from FreeBSD 9-Stable in an effort to evaluate the performance of our product, and perform an analysis and tracing of internal execution of our userland process. Some time ago I learned from Rui Paulo that this work was already done in FreeBSD 8.3 (r219107). To experiment, I installed FreeBSD 8.3-Release amd64, and recompiled the kernel and rebuilt the world per instructions provided here https://wiki.freebsd.org/Dtrace/ However, running the basic test in https://wiki.freebsd.org/DTrace/userlandfails in attempt to probe libc (or any system library) functions. (Note that the exact same issue happens when I manually backport userland support to FreeBSD 8.2-Release) # dtrace -s pid.d -c ./test test: failed to compile script pid.d: line 4: probe description pid2104:libc*:sleep:entry does not match any probes User signal 1 Similarly, probing stops and an application gets killed when its time to probe calls from ld-elf.so.1, libutil.so.8, libc.so.7, etc. Happens when you run: # dtrace -ln 'pid$target:::entry' -c date ID PROVIDER MODULE FUNCTION NAME 38195 pid2186 date usage entry 38196 pid2186 date badformat entry 38197 pid2186 date daysinmonth entry 38198 pid2186 date trans entry 38199 pid2186 date domktime entry 38200 pid2186 date adjhour entry 38201 pid2186 date adjyear entry 38202 pid2186 date adjmon entry 38203 pid2186 date adjday entry 38204 pid2186 date adjmin entry 38205 pid2186 date adjwday entry 38206 pid2186 date vary_destroy entry 38207 pid2186 date vary_append entry 38208 pid2186 date vary_apply entry 38209 pid2186 date netsettime entry 38210 pid2186 date _start entry 38211 pid2186 date main entry User signal 1 And, what is more strange, probing returns fails as well: # env DTRACE_DEBUG=1 dtrace -ln 'pid$target:::return' -c date Truncated debug output: ... libdtrace DEBUG: grabbed pid 2164 (cached) libdtrace DEBUG: date stret 0 0 0 0 libdtrace DEBUG: st_size of __do_global_dtors_aux is zero libdtrace DEBUG: st_size of frame_dummy is zero libdtrace DEBUG: st_size of __do_global_ctors_aux is zero libdtrace DEBUG: creating probe pid2164:date:usage:return libdtrace DEBUG: mr sparkle: Pread() failed libdtrace DEBUG: creating probe pid2164:date:badformat:return libdtrace DEBUG: mr sparkle: Pread() failed ... libdtrace DEBUG: creating probe pid2164:date:netsettime:return libdtrace DEBUG: mr sparkle: Pread() failed libdtrace DEBUG: creating probe pid2164:date:_start:return libdtrace DEBUG: mr sparkle: Pread() failed libdtrace DEBUG: creating probe pid2164:date:main:return libdtrace DEBUG: mr sparkle: Pread() failed libdtrace DEBUG: st_size of _fini is zero libdtrace DEBUG: st_size of _init is zero libdtrace DEBUG: st_size of _fini is zero libdtrace DEBUG: set context to pid2164:::return [0] prp=0x0 attr=[u/u/c] argc=0 libdtrace DEBUG: set label to libdtrace DEBUG: set label to ID PROVIDER MODULE FUNCTION NAME date: failed to match pid2164:::return: No probe matches description libdtrace DEBUG: killing pid 2164 User signal 1 Based on the email correspondence with Rui, he suspects that perhaps MFC wasn't done correctly, but currently doesn't have any idea of what is going on. I even tried FreeBSD 8-STABLE, and still same issue. I was hoping someone could shed some light and point me to a direction of how I can resolve this issue. I am open to any specific debugging suggestions you may have, and I will be glad to try them out to get you guys more information. I would very much appreciate any pointers. Thanks, -Akbar