From owner-freebsd-dtrace@FreeBSD.ORG Sun Oct 20 23:52:22 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D55116CD; Sun, 20 Oct 2013 23:52:22 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qe0-x231.google.com (mail-qe0-x231.google.com [IPv6:2607:f8b0:400d:c02::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8491A2A7E; Sun, 20 Oct 2013 23:52:22 +0000 (UTC) Received: by mail-qe0-f49.google.com with SMTP id a11so3353518qen.36 for ; Sun, 20 Oct 2013 16:52:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=aCL+35Gmqk7K9XNdpNirg8DsmUS9+hcWhLg2XaUDb2k=; b=p29TViDfoLiOM9pCbKCx6MEnC0RhoiY1EA+DdaffTYjXRTU/MKpMhJRMdeYcXqJvDB qEo5QKqHd2cmr5C1HkF4kHHjimhRPdOehKLTxSqQZyVinTFuw0cQba/ecNBxjFwDTU7K oaMqUrXfxoiFjg9j/nxTTuHwrKBOwoYNsh8pqszW6/N8xUkMLSZTykhvvDTieh0G1VhV MEEuEsQHU/lA9OyiC7R2/ir5CJKb0GAm+WbK9nkuhdJIOK2r73040++1TdZ8786y/ga2 MKWe+NTeH7xZAwNjrSkfbTQ+8tikRXuv86wuVoF3Ub6s7CWuWhcpyZPm/r+PYTKH6is5 +94A== X-Received: by 10.224.32.133 with SMTP id c5mr19937612qad.84.1382313141723; Sun, 20 Oct 2013 16:52:21 -0700 (PDT) Received: from charmander (24-212-218-13.cable.teksavvy.com. [24.212.218.13]) by mx.google.com with ESMTPSA id a5sm32154924qae.2.2013.10.20.16.52.20 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 20 Oct 2013 16:52:21 -0700 (PDT) Sender: Mark Johnston Date: Sun, 20 Oct 2013 19:52:16 -0400 From: Mark Johnston To: Veniamin Gvozdikov Subject: Re: Integration DTrace problems Message-ID: <20131020235216.GA13816@charmander> References: <6A174747-D855-481D-A191-67A2805BC9AE@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6A174747-D855-481D-A191-67A2805BC9AE@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Oct 2013 23:52:23 -0000 On Wed, Oct 09, 2013 at 01:54:14AM +0400, Veniamin Gvozdikov wrote: > Hello, > > I have problems with integration DTrace: > > * Wildcard bug > * USDT at runtime works only with probes with arguments less than 5 The patch below should partly address this issue (which exists for the pid provider too). On amd64, the first six arguments are taken out of the trap frame with (roughly) the following line: val = regs->r_rdi[argno] where 0 <= argno <= 5 is the argument index. This works when the six argument registers correspond to consecutive of struct reg, which is not the case on FreeBSD. This bug only comes up when argno == 5 though, since the first _five_ arguments are passed directly to dtrace_probe() by the fasttrap trap handler. Arguments of index > 5 come from the stack and may still be incorrect with this patch. I'm not yet sure what the problem is - if the arguments are (4-byte) ints, some of the values are wrong, but when I change the types to long, the values are correct (up to arg9 at least). > * USDT depended by base src because need dtrace.h although It exists on OSX and Oracle Linux > * Bug with providers position in D file with multi link dtrace objects > * Bug with not used probes when all providers unavailable if doesn't use in code > * Inconvenient toolchains (need to see DTrace on OSX) > > For more details go to link http://zlonet.ru/page/dtrace-integration-features/ . > > > Any idea? diff --git a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c index 65991af..8afc45a 100644 --- a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c +++ b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c @@ -271,8 +271,26 @@ fasttrap_anarg(struct reg *rp, int function_entry, int argno) * In 64-bit mode, the first six arguments are stored in * registers. */ - if (argno < 6) + if (argno < 6) { +#if !defined(sun) + switch (argno) { + case 0: + return (rp->r_rdi); + case 1: + return (rp->r_rsi); + case 2: + return (rp->r_rdx); + case 3: + return (rp->r_rcx); + case 4: + return (rp->r_r8); + case 5: + return (rp->r_r9); + } +#else return ((&rp->r_rdi)[argno]); +#endif + } stack = (uintptr_t *)rp->r_rsp; DTRACE_CPUFLAG_SET(CPU_DTRACE_NOFAULT); diff --git a/sys/cddl/dev/dtrace/amd64/dtrace_isa.c b/sys/cddl/dev/dtrace/amd64/dtrace_isa.c index 34d6f33..b82b77f 100644 --- a/sys/cddl/dev/dtrace/amd64/dtrace_isa.c +++ b/sys/cddl/dev/dtrace/amd64/dtrace_isa.c @@ -367,7 +367,31 @@ dtrace_getarg(int arg, int aframes) sizeof (uintptr_t)); if (arg <= inreg) { +#if !defined(sun) + switch (arg) { + case 0: + stack = (uintptr_t *)&rp->r_rdi; + break; + case 1: + stack = (uintptr_t *)&rp->r_rsi; + break; + case 2: + stack = (uintptr_t *)&rp->r_rdx; + break; + case 3: + stack = (uintptr_t *)&rp->r_rcx; + break; + case 4: + stack = (uintptr_t *)&rp->r_r8; + break; + case 5: + stack = (uintptr_t *)&rp->r_r9; + break; + } + arg = 0; +#else stack = (uintptr_t *)&rp->r_rdi; +#endif } else { stack = (uintptr_t *)(rp->r_rsp); arg -= inreg; From owner-freebsd-dtrace@FreeBSD.ORG Mon Oct 21 02:29:07 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7DC8E750 for ; Mon, 21 Oct 2013 02:29:07 +0000 (UTC) (envelope-from adam.leventhal@delphix.com) Received: from mail-ob0-x229.google.com (mail-ob0-x229.google.com [IPv6:2607:f8b0:4003:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 458CD21ED for ; Mon, 21 Oct 2013 02:29:07 +0000 (UTC) Received: by mail-ob0-f169.google.com with SMTP id wp4so5266274obc.0 for ; Sun, 20 Oct 2013 19:29:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=jf8kENLxhmO3Bb34PIagdtB77TsZOQYUGAF7e9cjxNw=; b=ceXoC7uxmw18YK2KRz+x0LlkZYShdSHnZzouT12sWUkCWZCt1Iq6ye34ihnuW+oqOy CwzRD9xTdLMBUHSpYs5479DSKgOKX2diRURMSCxJn4dixLb2WP9KBkniWNmKtEGscaUT IqwYhUtbNZbG42zxnY20bCsWm0rKvOwma58ak= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=jf8kENLxhmO3Bb34PIagdtB77TsZOQYUGAF7e9cjxNw=; b=GdBSMd1gnJmycENemcZiG3E5wIEbhVJ6wkNz2inLP7BkWGlqNLTzOoiE3wTh/J2b6v IA1knYPzdOkgquiMBcgSx9sYmrzK289tZf0PMeN9N4gKVqaGOoMCxfK4KK/TvYJjh0+D dwrNlec07zS2a2Z2pBZJVS458MG5zddDrE7e1elvjOASegyCnNlAbZmQi1MB/MwM7uCr oAIyhyMTd8+9QD+JBq4tsg7+aWMCom9d9JANh2LwDfDHSJq4wXF5VFY9ojxj56VEnUKZ is2MAKjZrf8LtIDv5QA1aSVltMOKu7c5yHwoNcLiYo6Vz89fXJdhFkCg0jwef6ZYbpg/ 2yqA== X-Gm-Message-State: ALoCoQldoaf9OTlTl/9ZML0DKMP0WRrzd47KUrpLMMxx4k+ohW9LFVizu5IQBUQz7R8CBykw3Vsi MIME-Version: 1.0 X-Received: by 10.60.96.169 with SMTP id dt9mr5226066oeb.27.1382322546280; Sun, 20 Oct 2013 19:29:06 -0700 (PDT) Received: by 10.76.92.67 with HTTP; Sun, 20 Oct 2013 19:29:06 -0700 (PDT) In-Reply-To: <20131020235216.GA13816@charmander> References: <6A174747-D855-481D-A191-67A2805BC9AE@freebsd.org> <20131020235216.GA13816@charmander> Date: Sun, 20 Oct 2013 21:29:06 -0500 Message-ID: Subject: Re: Integration DTrace problems From: Adam Leventhal To: Mark Johnston Content-Type: text/plain; charset=ISO-8859-1 Cc: Veniamin Gvozdikov , freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Oct 2013 02:29:07 -0000 Hey Mark, I'd go so far as to suggest you remove the #ifdef sun stuff and just have your new logic replace what was there. As the original author of that code (I think), it is more kludgy than clever in retrospect -- as so many things are... Thanks. Adam On Sun, Oct 20, 2013 at 6:52 PM, Mark Johnston wrote: > On Wed, Oct 09, 2013 at 01:54:14AM +0400, Veniamin Gvozdikov wrote: >> Hello, >> >> I have problems with integration DTrace: >> >> * Wildcard bug >> * USDT at runtime works only with probes with arguments less than 5 > > The patch below should partly address this issue (which exists for the > pid provider too). On amd64, the first six arguments are taken out of the > trap frame with (roughly) the following line: > > val = regs->r_rdi[argno] > > where 0 <= argno <= 5 is the argument index. This works when the six > argument registers correspond to consecutive of struct reg, which is not > the case on FreeBSD. This bug only comes up when argno == 5 though, > since the first _five_ arguments are passed directly to dtrace_probe() by > the fasttrap trap handler. > > Arguments of index > 5 come from the stack and may still be incorrect > with this patch. I'm not yet sure what the problem is - if the arguments > are (4-byte) ints, some of the values are wrong, but when I change the > types to long, the values are correct (up to arg9 at least). > >> * USDT depended by base src because need dtrace.h although It exists on OSX and Oracle Linux >> * Bug with providers position in D file with multi link dtrace objects >> * Bug with not used probes when all providers unavailable if doesn't use in code >> * Inconvenient toolchains (need to see DTrace on OSX) >> >> For more details go to link http://zlonet.ru/page/dtrace-integration-features/ . >> >> >> Any idea? > > diff --git a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > index 65991af..8afc45a 100644 > --- a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > +++ b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > @@ -271,8 +271,26 @@ fasttrap_anarg(struct reg *rp, int function_entry, int argno) > * In 64-bit mode, the first six arguments are stored in > * registers. > */ > - if (argno < 6) > + if (argno < 6) { > +#if !defined(sun) > + switch (argno) { > + case 0: > + return (rp->r_rdi); > + case 1: > + return (rp->r_rsi); > + case 2: > + return (rp->r_rdx); > + case 3: > + return (rp->r_rcx); > + case 4: > + return (rp->r_r8); > + case 5: > + return (rp->r_r9); > + } > +#else > return ((&rp->r_rdi)[argno]); > +#endif > + } > > stack = (uintptr_t *)rp->r_rsp; > DTRACE_CPUFLAG_SET(CPU_DTRACE_NOFAULT); > diff --git a/sys/cddl/dev/dtrace/amd64/dtrace_isa.c b/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > index 34d6f33..b82b77f 100644 > --- a/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > +++ b/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > @@ -367,7 +367,31 @@ dtrace_getarg(int arg, int aframes) > sizeof (uintptr_t)); > > if (arg <= inreg) { > +#if !defined(sun) > + switch (arg) { > + case 0: > + stack = (uintptr_t *)&rp->r_rdi; > + break; > + case 1: > + stack = (uintptr_t *)&rp->r_rsi; > + break; > + case 2: > + stack = (uintptr_t *)&rp->r_rdx; > + break; > + case 3: > + stack = (uintptr_t *)&rp->r_rcx; > + break; > + case 4: > + stack = (uintptr_t *)&rp->r_r8; > + break; > + case 5: > + stack = (uintptr_t *)&rp->r_r9; > + break; > + } > + arg = 0; > +#else > stack = (uintptr_t *)&rp->r_rdi; > +#endif > } else { > stack = (uintptr_t *)(rp->r_rsp); > arg -= inreg; > _______________________________________________ > freebsd-dtrace@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-dtrace > To unsubscribe, send any mail to "freebsd-dtrace-unsubscribe@freebsd.org" -- Adam Leventhal CTO, Delphix http://blog.delphix.com/ahl From owner-freebsd-dtrace@FreeBSD.ORG Mon Oct 21 03:49:39 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D8457883 for ; Mon, 21 Oct 2013 03:49:39 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A91B42698 for ; Mon, 21 Oct 2013 03:49:39 +0000 (UTC) Received: by mail-ie0-f174.google.com with SMTP id qd12so10545854ieb.5 for ; Sun, 20 Oct 2013 20:49:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=AGaonP/RH6U9hBMmzVsjF3MXIm1hM52aUhVBI9RVVf8=; b=V3A1XfTJjTiEEG5aCb6EPldhVEqyH25Gk+BTocayajLgan91+heYHAg2gkkJYYVR0G 1vK0poD88DLWR5NHZRVkKNydnxL8Jidduo3wj5FMPD24FtJ8ATesT817JNTQzHFi6EFT hGw/vHTkAUpbwAIg2J1mpgfIMlDXSXgBTeKSJYpNe9b0hFkdbJlZ7itVc+oW6J98HGkB SMm32uyHWtlyRvYE8a+myA3f81752ACAh4emeKRpel9m6SmfXR0rN7VlmlIirWd7BYZO kJwWHV5DomPqtfi+Vzxq0vBc0YLNbyXXVHatxN/35zX+lXjDbnImx8N5aQCic+qtBgsB Ca4A== X-Received: by 10.50.225.39 with SMTP id rh7mr8035877igc.10.1382327378970; Sun, 20 Oct 2013 20:49:38 -0700 (PDT) Received: from charmander (24-212-218-13.cable.teksavvy.com. [24.212.218.13]) by mx.google.com with ESMTPSA id x6sm28253552igb.3.2013.10.20.20.49.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 20 Oct 2013 20:49:38 -0700 (PDT) Sender: Mark Johnston Date: Sun, 20 Oct 2013 23:48:59 -0400 From: Mark Johnston To: John-Mark Gurney Subject: Re: new issues w/ dtrace aborting... Message-ID: <20131021034859.GA2402@charmander> References: <20131017200658.GG56872@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131017200658.GG56872@funkthat.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Oct 2013 03:49:39 -0000 On Thu, Oct 17, 2013 at 01:06:58PM -0700, John-Mark Gurney wrote: > I'm see this failure which is reproducable: > # dtrace -s ./disklatencycmd.d -x evaltime=postinit -c ./catall > Tracing... Hit Ctrl-C to end. > > Now, I have spent some time trying to debug this error and have > gotten a stack trace: > #0 dt_divide_128 (dividend=0x7fffffffd240, divisor=0, quotient=0x7fffffffd240) > at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c:221 > #1 0x0000000800a8d343 in dt_stddev (data=0x802bb3e48, normal=1) > at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c:372 > #2 0x0000000800a9549a in dt_aggregate_valcmp (lhs=, > rhs=) > at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:120 > #3 0x0000000800a93821 in dt_aggregate_varvalcmp (lhs=0x804679240, > rhs=0x804679260) > at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:950 > #4 0x0000000801a3d6aa in qsort () from /lib/libc.so.7 > #5 0x0000000800a93c88 in dt_aggregate_walk_sorted (dtp=0x80281d000, > func=0x800a8f7d0 , arg=0x7fffffffd498, > sfunc=) > at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:1275 > #6 0x0000000800a951c6 in dtrace_aggregate_print (dtp=0x80281d000, > fp=, func=) > at /usr/src/cddl/lib/libdtrace/../../../cddl/contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:1880 > #7 0x000000000040387f in main (argc=, > argv=0x7fffffffd700) > at /usr/src/cddl/usr.sbin/dtrace/../../../cddl/contrib/opensolaris/cmd/dtrace/dtrace.c:1921 > > Now the weird part is that between frame 3 and frame 0, the address of > lhs/data changes, yet if you look at the code, it is passing the pointer > straight through w/o modification... The value pointed to by lhs is > valid and non-zero, but in frame 0 the different pointer is now zero.. > > The script catall: > #!/bin/sh - > > #since dtrace has issues: > find /mnt -type f -exec cat {} + > /dev/null > > The directory /mnt has a FS that only contains a recent export of HEAD.. > I also umount/mount before each run to make sure the disk cache is clear, > otherwise it's possible that all the data will be cached in memory, and > not perform any io... I can reproduce this by just running it on my laptop and killing it with ctrl-C. I'll work on figuring it out this week. Thanks for reporting it. > > It looks like it's an issue w/ clear(@stddev) as this is the script I've > reduced it to reproduce the failiure: > #pragma D option quiet > #pragma D option dynvarsize=16m > > io:::start > { > start_time[arg0] = timestamp; > } > > io:::done > /this->start = start_time[arg0]/ > { > this->delta = (timestamp - this->start) / 1000; > @stddev[args[1]->device_name, args[1]->unit_number] = stddev(this->delta > ); > start_time[arg0] = 0; > } > > dtrace:::END > { > clear(@stddev); > } > > Obviously for this to be useful, you'd print out the stddev, but the > abort happens either way... > > Thanks. > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > _______________________________________________ > freebsd-dtrace@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-dtrace > To unsubscribe, send any mail to "freebsd-dtrace-unsubscribe@freebsd.org" From owner-freebsd-dtrace@FreeBSD.ORG Mon Oct 21 04:20:31 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C283B2A7; Mon, 21 Oct 2013 04:20:31 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-ie0-x234.google.com (mail-ie0-x234.google.com [IPv6:2607:f8b0:4001:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 876E62889; Mon, 21 Oct 2013 04:20:31 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id e14so10335308iej.39 for ; Sun, 20 Oct 2013 21:20:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=1sBjIddx+gmdKFLBA9UvcOIpL7iyC18ZDGrl7Y+YLX4=; b=cmqoGDxBfej4/mNyqMnna9iuYXHgFsOiapF5iW7N2ku1jo3eBRon9dsRlpIJu5tEnf 0ymd77wVeCscgZ3HK1TMT/pbcTkbM/3QgLc0kdIQUPnqVFTM+9uQKtOwunMAF6P9HrTj H7ws6zWyw5Xh1PcZaRb6GwU2uqZrnRE3zZ78UUSK3qA1SlnAmGQSCWxwQTlghdp5QRtH sFPSVgidN/9smgkXvPVcilbR5xV1dOMIoznVyL6TRHNuJNcXEZolfVSkeA5iMltFFwUQ H2v2qBAMKCv9vrCGiyzX7occtZWlUov2uYpMPOpb1eNCCZNL5yJG6llHhjlWjhG/cVNE C07w== X-Received: by 10.50.153.50 with SMTP id vd18mr7934555igb.6.1382329230420; Sun, 20 Oct 2013 21:20:30 -0700 (PDT) Received: from raichu (24-212-218-13.cable.teksavvy.com. [24.212.218.13]) by mx.google.com with ESMTPSA id m1sm28358344igj.10.2013.10.20.21.20.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 20 Oct 2013 21:20:29 -0700 (PDT) Sender: Mark Johnston Date: Mon, 21 Oct 2013 00:20:27 -0400 From: Mark Johnston To: Adam Leventhal Subject: Re: Integration DTrace problems Message-ID: <20131021042027.GC13315@raichu> References: <6A174747-D855-481D-A191-67A2805BC9AE@freebsd.org> <20131020235216.GA13816@charmander> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Veniamin Gvozdikov , freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Oct 2013 04:20:31 -0000 On Sun, Oct 20, 2013 at 09:29:06PM -0500, Adam Leventhal wrote: > Hey Mark, > > I'd go so far as to suggest you remove the #ifdef sun stuff and just > have your new logic replace what was there. As the original author of > that code (I think), it is more kludgy than clever in retrospect -- as > so many things are... Fair enough. I removed the #ifdefs and committed the change as r256822. Thanks! -Mark > > Thanks. > > Adam > > On Sun, Oct 20, 2013 at 6:52 PM, Mark Johnston wrote: > > On Wed, Oct 09, 2013 at 01:54:14AM +0400, Veniamin Gvozdikov wrote: > >> Hello, > >> > >> I have problems with integration DTrace: > >> > >> * Wildcard bug > >> * USDT at runtime works only with probes with arguments less than 5 > > > > The patch below should partly address this issue (which exists for the > > pid provider too). On amd64, the first six arguments are taken out of the > > trap frame with (roughly) the following line: > > > > val = regs->r_rdi[argno] > > > > where 0 <= argno <= 5 is the argument index. This works when the six > > argument registers correspond to consecutive of struct reg, which is not > > the case on FreeBSD. This bug only comes up when argno == 5 though, > > since the first _five_ arguments are passed directly to dtrace_probe() by > > the fasttrap trap handler. > > > > Arguments of index > 5 come from the stack and may still be incorrect > > with this patch. I'm not yet sure what the problem is - if the arguments > > are (4-byte) ints, some of the values are wrong, but when I change the > > types to long, the values are correct (up to arg9 at least). > > > >> * USDT depended by base src because need dtrace.h although It exists on OSX and Oracle Linux > >> * Bug with providers position in D file with multi link dtrace objects > >> * Bug with not used probes when all providers unavailable if doesn't use in code > >> * Inconvenient toolchains (need to see DTrace on OSX) > >> > >> For more details go to link http://zlonet.ru/page/dtrace-integration-features/ . > >> > >> > >> Any idea? > > > > diff --git a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > > index 65991af..8afc45a 100644 > > --- a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > > +++ b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > > @@ -271,8 +271,26 @@ fasttrap_anarg(struct reg *rp, int function_entry, int argno) > > * In 64-bit mode, the first six arguments are stored in > > * registers. > > */ > > - if (argno < 6) > > + if (argno < 6) { > > +#if !defined(sun) > > + switch (argno) { > > + case 0: > > + return (rp->r_rdi); > > + case 1: > > + return (rp->r_rsi); > > + case 2: > > + return (rp->r_rdx); > > + case 3: > > + return (rp->r_rcx); > > + case 4: > > + return (rp->r_r8); > > + case 5: > > + return (rp->r_r9); > > + } > > +#else > > return ((&rp->r_rdi)[argno]); > > +#endif > > + } > > > > stack = (uintptr_t *)rp->r_rsp; > > DTRACE_CPUFLAG_SET(CPU_DTRACE_NOFAULT); > > diff --git a/sys/cddl/dev/dtrace/amd64/dtrace_isa.c b/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > > index 34d6f33..b82b77f 100644 > > --- a/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > > +++ b/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > > @@ -367,7 +367,31 @@ dtrace_getarg(int arg, int aframes) > > sizeof (uintptr_t)); > > > > if (arg <= inreg) { > > +#if !defined(sun) > > + switch (arg) { > > + case 0: > > + stack = (uintptr_t *)&rp->r_rdi; > > + break; > > + case 1: > > + stack = (uintptr_t *)&rp->r_rsi; > > + break; > > + case 2: > > + stack = (uintptr_t *)&rp->r_rdx; > > + break; > > + case 3: > > + stack = (uintptr_t *)&rp->r_rcx; > > + break; > > + case 4: > > + stack = (uintptr_t *)&rp->r_r8; > > + break; > > + case 5: > > + stack = (uintptr_t *)&rp->r_r9; > > + break; > > + } > > + arg = 0; > > +#else > > stack = (uintptr_t *)&rp->r_rdi; > > +#endif > > } else { > > stack = (uintptr_t *)(rp->r_rsp); > > arg -= inreg; > > _______________________________________________ > > freebsd-dtrace@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-dtrace > > To unsubscribe, send any mail to "freebsd-dtrace-unsubscribe@freebsd.org" > > > > -- > Adam Leventhal > CTO, Delphix > http://blog.delphix.com/ahl From owner-freebsd-dtrace@FreeBSD.ORG Wed Oct 23 03:13:51 2013 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0D31B1B5; Wed, 23 Oct 2013 03:13:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C65562C3F; Wed, 23 Oct 2013 03:13:50 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id aq17so350083iec.38 for ; Tue, 22 Oct 2013 20:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=T78s73LimSO7Ml6VHpzxdvMnCsf6VvWonHto4bHAAkc=; b=snbGgBtfPCAvhvKsjd5hCmCY3j8YSPkRuiFU7tNFvYyTQBUM/W8DdWafjfzdQEYKyI wbatawToxxEDLr9bnbAaKWKiZggKFZjTQaTvUmJQFTgWtfnZdCM0p6gRPdxtzePzxhYy w/E4UR4VLAMLCj8YMHfZYTH5qeoM5PNw4ywh068I8wIcNIQ+R8OsKhltvdD81Mr+hQhn cppSx+/ZTlgCqnEDtgyeIVGF6zizs6S/2Bim3R3QADnAR81FOV9sgV5gzv+v+ctorO+Q AuXztlWtSIIaYPqxEdaWpnr+gzGXg2HaJ3XPFgQzCAs4U8PltyRvOtTsrrihBL+lpfGL 916g== X-Received: by 10.50.106.20 with SMTP id gq20mr61611igb.36.1382498030107; Tue, 22 Oct 2013 20:13:50 -0700 (PDT) Received: from raichu (24-212-218-13.cable.teksavvy.com. [24.212.218.13]) by mx.google.com with ESMTPSA id yt10sm5913570igb.9.2013.10.22.20.13.48 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 22 Oct 2013 20:13:49 -0700 (PDT) Sender: Mark Johnston Date: Tue, 22 Oct 2013 23:13:44 -0400 From: Mark Johnston To: freebsd-dtrace@freebsd.org Subject: Re: Integration DTrace problems Message-ID: <20131023031344.GA8033@raichu> References: <6A174747-D855-481D-A191-67A2805BC9AE@freebsd.org> <20131020235216.GA13816@charmander> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131020235216.GA13816@charmander> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Veniamin Gvozdikov X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Oct 2013 03:13:51 -0000 On Sun, Oct 20, 2013 at 07:52:16PM -0400, Mark Johnston wrote: > On Wed, Oct 09, 2013 at 01:54:14AM +0400, Veniamin Gvozdikov wrote: > > Hello, > > > > I have problems with integration DTrace: > > > > * Wildcard bug > > * USDT at runtime works only with probes with arguments less than 5 > > ... > > Arguments of index > 5 come from the stack and may still be incorrect > with this patch. I'm not yet sure what the problem is - if the arguments > are (4-byte) ints, some of the values are wrong, but when I change the > types to long, the values are correct (up to arg9 at least). This comment can be ignored - I had forgotten that the argN variables are always 64 bits wide and so may give incorrect results when trying to print a 32-bit function argument. Switching to args[N] gives me correct results. So with r256822, argument fetching seems to be working properly with USDT and PID probes. :) -Mark > > > * USDT depended by base src because need dtrace.h although It exists on OSX and Oracle Linux > > * Bug with providers position in D file with multi link dtrace objects > > * Bug with not used probes when all providers unavailable if doesn't use in code > > * Inconvenient toolchains (need to see DTrace on OSX) > > > > For more details go to link http://zlonet.ru/page/dtrace-integration-features/ . > > > > > > Any idea? > > diff --git a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > index 65991af..8afc45a 100644 > --- a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > +++ b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c > @@ -271,8 +271,26 @@ fasttrap_anarg(struct reg *rp, int function_entry, int argno) > * In 64-bit mode, the first six arguments are stored in > * registers. > */ > - if (argno < 6) > + if (argno < 6) { > +#if !defined(sun) > + switch (argno) { > + case 0: > + return (rp->r_rdi); > + case 1: > + return (rp->r_rsi); > + case 2: > + return (rp->r_rdx); > + case 3: > + return (rp->r_rcx); > + case 4: > + return (rp->r_r8); > + case 5: > + return (rp->r_r9); > + } > +#else > return ((&rp->r_rdi)[argno]); > +#endif > + } > > stack = (uintptr_t *)rp->r_rsp; > DTRACE_CPUFLAG_SET(CPU_DTRACE_NOFAULT); > diff --git a/sys/cddl/dev/dtrace/amd64/dtrace_isa.c b/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > index 34d6f33..b82b77f 100644 > --- a/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > +++ b/sys/cddl/dev/dtrace/amd64/dtrace_isa.c > @@ -367,7 +367,31 @@ dtrace_getarg(int arg, int aframes) > sizeof (uintptr_t)); > > if (arg <= inreg) { > +#if !defined(sun) > + switch (arg) { > + case 0: > + stack = (uintptr_t *)&rp->r_rdi; > + break; > + case 1: > + stack = (uintptr_t *)&rp->r_rsi; > + break; > + case 2: > + stack = (uintptr_t *)&rp->r_rdx; > + break; > + case 3: > + stack = (uintptr_t *)&rp->r_rcx; > + break; > + case 4: > + stack = (uintptr_t *)&rp->r_r8; > + break; > + case 5: > + stack = (uintptr_t *)&rp->r_r9; > + break; > + } > + arg = 0; > +#else > stack = (uintptr_t *)&rp->r_rdi; > +#endif > } else { > stack = (uintptr_t *)(rp->r_rsp); > arg -= inreg; From owner-freebsd-dtrace@FreeBSD.ORG Wed Oct 23 20:30:18 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9F6574CE for ; Wed, 23 Oct 2013 20:30:18 +0000 (UTC) (envelope-from symbolics@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 36CB02E5B for ; Wed, 23 Oct 2013 20:30:18 +0000 (UTC) Received: from lemon ([80.7.17.14]) by mail.gmx.com (mrgmx002) with ESMTPSA (Nemesis) id 0MJSLz-1VXiQM3RNb-00342w for ; Wed, 23 Oct 2013 22:30:09 +0200 Received: by lemon (Postfix, from userid 1001) id 2F321EB332; Wed, 23 Oct 2013 21:30:09 +0100 (BST) Date: Wed, 23 Oct 2013 21:30:09 +0100 From: symbolics@gmx.com To: dtrace@freebsd.org Subject: Firefox crash during dtrace attach under -CURRENT Message-ID: <20131023203009.GA92945@lemon> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Provags-ID: V03:K0:qkPnBpJJImUiAODzVK0Q7sJa6KZ7Y6VF2MubjvFPfDaHyYzgJv/ W3gWZKR1J14apTp3u5YS0j4YFcX0H36v+gyNSxrhtcpcTPqlNU/Gcy+reTpe3NbydL13BmY FiEOph7j+zb34BRqnuQIinKje4RwncMIzG5/rzFWKtJNyO41ZC5f+HY8b3rjicYjjQ7ASJ7 gD5ExSgUf9LgMCgjhepbQ== X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Oct 2013 20:30:18 -0000 Hi, http://dtrace.org/blogs/brendan/2011/02/11/dtrace-pid-provider-arguments/ I tried to follow some of the examples but I crash the Firefox process each time. Sometimes DTrace manages to collect a little data before the death. # dtrace -n 'pid$target::malloc:entry { trace(arg0); }' -p 92974 dtrace: ERROR: gelf_getehdr() failed: No error: 0 [ ~30 lines ] dtrace: ERROR: gelf_getehdr() failed: No error: 0 dtrace: description 'pid$target::malloc:entry ' matched 1 probe CPU ID FUNCTION:NAME 2 53672 __malloc:entry 48 2 53672 __malloc:entry 106 2 53672 __malloc:entry 168 2 53672 __malloc:entry 57 2 53672 __malloc:entry 32 1 53672 __malloc:entry 32 2 53672 __malloc:entry 22 2 53672 __malloc:entry 22 2 53672 __malloc:entry 23 2 53672 __malloc:entry 23 2 53672 __malloc:entry 16 2 53672 __malloc:entry 14 2 53672 __malloc:entry 40 2 53672 __malloc:entry 36 2 53672 __malloc:entry 104 2 53672 __malloc:entry 160 2 53672 __malloc:entry 24 2 53672 __malloc:entry 24 2 53672 __malloc:entry 14 2 53672 __malloc:entry 14 2 53672 __malloc:entry 37 2 53672 __malloc:entry 37 2 53672 __malloc:entry 33 2 53672 __malloc:entry 33 2 53672 __malloc:entry 16 2 53672 __malloc:entry 14 2 53672 __malloc:entry 40 2 53672 __malloc:entry 36 2 53672 __malloc:entry 104 2 53672 __malloc:entry 160 2 53672 __malloc:entry 29 2 53672 __malloc:entry 29 2 53672 __malloc:entry 80 dtrace: processing aborted: No error: 0 >From the Firefox core: #0 StackShape (this=0x8150154b1, shape=@0x8150154d1) at Barrier.h:156 156 Barrier.h: No such file or directory. in Barrier.h [New Thread 816d60400 (LWP 100253/mozStorage #6)] [New Thread 815de2c00 (LWP 100251/mozStorage #5)] [New Thread 81b008c00 (LWP 100249/DOM Worker)] [New Thread 816b5f400 (LWP 100248/DNS Resolver #3)] [New Thread 81ace7800 (LWP 100247/ImageDecoder #2)] [New Thread 81a405800 (LWP 100246/DNS Resolver #2)] [New Thread 815fcc800 (LWP 100245/Cache Deleter)] [New Thread 81905b400 (LWP 100244/DNS Resolver #1)] [New Thread 818451c00 (LWP 100243/Proxy Resolution)] [New Thread 818450c00 (LWP 100242/mozStorage #4)] [New Thread 81841fc00 (LWP 100241/URL Classifier)] [New Thread 8175ecc00 (LWP 100240/Cert Verify)] [New Thread 817dc0400 (LWP 100238/mozStorage #3)] [New Thread 817dbbc00 (LWP 100237/localStorage DB)] [New Thread 815bc9400 (LWP 100236/mozStorage #2)] [New Thread 815bcb400 (LWP 100235/mozStorage #1)] [New Thread 816b58000 (LWP 100232/ImageDecoder #1)] [New Thread 801a04000 (LWP 100231/HTML5 Parser)] [New Thread 815fc6800 (LWP 100230/Cache I/O)] [New Thread 815133400 (LWP 100228/DOM Worker)] [New Thread 813989800 (LWP 100227/Timer)] [New Thread 813845800 (LWP 100226/Hang Monitor)] [New Thread 813843400 (LWP 100218/JS Watchdog)] [New Thread 813842c00 (LWP 100213/firefox)] [New Thread 813842800 (LWP 100212/JS GC Helper)] [New Thread 813842400 (LWP 100211/Socket Thread)] [New Thread 801a09400 (LWP 100206/Gecko_IOThread)] [New Thread 801a02400 (LWP 100182/firefox)] # (gdb) bt full #0 StackShape (this=0x8150154b1, shape=@0x8150154d1) at Barrier.h:156 No locals. #1 0x0000000804a6bb11 in js::PropertyTree::insertChild (this=, cx=0x813834a00, parent=0x815987830, child=0xdadada0000006500) at jspropertytree.cpp:52 kidp = (js::KidsPointer *) 0xdadada0000006400 Previous frame inner to this frame (corrupt stack?) A different core gives a different stack trace. These were the only two I saw. #0 thr_kill () at thr_kill.S:3 3 thr_kill.S: No such file or directory. in thr_kill.S [New Thread 816fd8000 (LWP 100287/mozStorage #7)] [New Thread 815fc4c00 (LWP 100286/mozStorage #6)] [New Thread 81b476800 (LWP 100284/DOM Worker)] [New Thread 81903c800 (LWP 100279/Proxy Resolution)] [New Thread 81903bc00 (LWP 100278/mozStorage #5)] [New Thread 81856fc00 (LWP 100277/URL Classifier)] [New Thread 81856ac00 (LWP 100276/Cert Verify)] [New Thread 816fda800 (LWP 100275/mozStorage #4)] [New Thread 817dd9800 (LWP 100274/mozStorage #3)] [New Thread 817dd6800 (LWP 100273/localStorage DB)] [New Thread 816a70400 (LWP 100272/mozStorage #2)] [New Thread 816f4b800 (LWP 100271/mozStorage #1)] [New Thread 801a04000 (LWP 100266/HTML5 Parser)] [New Thread 815fc5800 (LWP 100265/Cache I/O)] [New Thread 815133400 (LWP 100263/DOM Worker)] [New Thread 813989800 (LWP 100262/Timer)] [New Thread 813845800 (LWP 100261/Hang Monitor)] [New Thread 813843400 (LWP 100260/JS Watchdog)] [New Thread 813842c00 (LWP 100259/firefox)] [New Thread 813842800 (LWP 100258/JS GC Helper)] [New Thread 813842400 (LWP 100257/Socket Thread)] [New Thread 801a09400 (LWP 100256/Gecko_IOThread)] [New Thread 801a02400 (LWP 100177/firefox)] (gdb) bt #0 thr_kill () at thr_kill.S:3 #1 0x00000008024e187a in nsProfileLock::FatalSignalHandler (signo=11, info=, context=) at nsProfileLock.cpp:180 #2 0x000000080113f3c6 in handle_signal (actp=, sig=11, info=0x7fffffffc270, ucp=0x7fffffffbf00) at /usr/home/dm/git/freebsd/lib/libthr/thread/thr_sig.c:237 #3 0x000000080113efc9 in thr_sighandler (sig=11, info=, _ucp=) at /usr/home/dm/git/freebsd/lib/libthr/thread/thr_sig.c:182 #4 0x00007ffffffff003 in ?? () #5 0x000000080113eeb0 in sigaction () at /usr/home/dm/git/freebsd/lib/libthr/thread/thr_sig.c:566 Previous frame inner to this frame (corrupt stack?) Current language: auto; currently asm Is this a known problem or should I send a PR? --sym From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 24 02:59:09 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 49EC2E96 for ; Thu, 24 Oct 2013 02:59:09 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qa0-x235.google.com (mail-qa0-x235.google.com [IPv6:2607:f8b0:400d:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0C2132320 for ; Thu, 24 Oct 2013 02:59:08 +0000 (UTC) Received: by mail-qa0-f53.google.com with SMTP id k15so1047999qaq.5 for ; Wed, 23 Oct 2013 19:59:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=i7BE0nbIKNPmCaeN5gVWISSLN62Dox2gMORfvdO0Q84=; b=vY0zFZWW45f567mULGgzsB40ejYER9DIORomgD3HoY4N8f2RdalcvVJ+MUdmLjhg9I /0Y8gkG1K1jMNSpZPJqFJa7qZJ9G7qDchkgkTcU4tgtXdRZF1GTIPr2t8QJp7JLqL/66 +OSfm6OsGlMgrbV3tOXm2VSqrueFg5uiLueNqckQOfLr+wCH9FAcTtn+RGI8cspCfKPg 5Lfex7OspoVDjIci99Ejqwqf5h6IncPxBQDNT/E2e8Kv3gkmuW/RQiQJBnES5+CBwrvq ubFHl2NQaw9fxju8VDPORX0b0r+kqWxosHibH3whQrvtDkiVp2XkxvDJknE9GNMKk1+F UDRg== X-Received: by 10.49.62.137 with SMTP id y9mr469445qer.59.1382583548273; Wed, 23 Oct 2013 19:59:08 -0700 (PDT) Received: from charmander (24-212-218-13.cable.teksavvy.com. [24.212.218.13]) by mx.google.com with ESMTPSA id d7sm3080813qas.10.2013.10.23.19.59.07 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 23 Oct 2013 19:59:07 -0700 (PDT) Sender: Mark Johnston Date: Wed, 23 Oct 2013 22:59:02 -0400 From: Mark Johnston To: symbolics@gmx.com Subject: Re: Firefox crash during dtrace attach under -CURRENT Message-ID: <20131024025902.GA2286@charmander> References: <20131023203009.GA92945@lemon> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131023203009.GA92945@lemon> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 02:59:09 -0000 On Wed, Oct 23, 2013 at 09:30:09PM +0100, symbolics@gmx.com wrote: > Hi, > > http://dtrace.org/blogs/brendan/2011/02/11/dtrace-pid-provider-arguments/ > > I tried to follow some of the examples but I crash the Firefox process > each time. Sometimes DTrace manages to collect a little data before the > death. > > [...] > > Is this a known problem or should I send a PR? Thanks for reporting this: I was able to reproduce the crash and managed to find a nasty pair of bugs. Could you test the patch below and let me know if it fixes the problem for you as well? If you see more crashes, please include the backtrace and signo from gdb again; it would likely be a different problem that needs to be debugged and fixed separately. For anyone interested, the bug is that fasttrap's ebp push instruction emulation code is just wrong: it's supposed to save %rbp at %rsp - 8. But instead it tries to save %rsp at %rsp - 8, and also reverses the uaddr/kaddr arguments to copyout(), resulting in strange crashes. I managed to narrow in on the problem with a test program that prints %rbp immediately before and after a tracepoint. Can anyone review this diff? I'd like to check it in soon, assuming that I haven't also made a mistake somewhere. :) Thanks, -Mark diff --git a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c index 8b5ce9f..bb5c9af 100644 --- a/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c +++ b/sys/cddl/contrib/opensolaris/uts/intel/dtrace/fasttrap_isa.c @@ -1399,12 +1399,12 @@ fasttrap_pid_probe(struct reg *rp) #ifdef __amd64 if (p->p_model == DATAMODEL_NATIVE) { addr = rp->r_rsp - sizeof (uintptr_t); - ret = fasttrap_sulword((void *)addr, &rp->r_rsp); + ret = fasttrap_sulword(&rp->r_rbp, (void *)addr); } else { #endif #ifdef __i386__ addr = rp->r_rsp - sizeof (uint32_t); - ret = fasttrap_suword32((void *)addr, &rp->r_rsp); + ret = fasttrap_suword32(&rp->r_rbp, (void *)addr); #endif #ifdef __amd64 } From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 24 10:21:15 2013 Return-Path: Delivered-To: dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5E11C258 for ; Thu, 24 Oct 2013 10:21:15 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 938F32FC4 for ; Thu, 24 Oct 2013 10:21:14 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA26442 for ; Thu, 24 Oct 2013 13:21:13 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1VZI2S-0005wS-LY for dtrace@FreeBSD.org; Thu, 24 Oct 2013 13:21:12 +0300 Message-ID: <5268F461.7080504@FreeBSD.org> Date: Thu, 24 Oct 2013 13:20:17 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: dtrace@FreeBSD.org Subject: "unstable" sdt probes X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=X-VIET-VPS Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 10:21:15 -0000 Can our SDT (for kernel) implementation support probes with unspecified types for arguments? I would like to get DTRACE_PROBE*() which can be found in the code with OpenSolaris origins (e.g. ZFS) to work. With minimal efforts :-) At present sys/sdt.h provides DTRACE_PROBE* only for userland implementation. Thanks! -- Andriy Gapon From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 24 11:06:45 2013 Return-Path: Delivered-To: dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 742674F9 for ; Thu, 24 Oct 2013 11:06:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id C9D0223E6 for ; Thu, 24 Oct 2013 11:06:44 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA26992 for ; Thu, 24 Oct 2013 14:06:43 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1VZIkV-0005zp-4J for dtrace@FreeBSD.org; Thu, 24 Oct 2013 14:06:43 +0300 Message-ID: <5268FF06.2000304@FreeBSD.org> Date: Thu, 24 Oct 2013 14:05:42 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: dtrace@FreeBSD.org Subject: lockstat X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=X-VIET-VPS Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 11:06:45 -0000 I see that there is a certain discrepancy between what userland lockstat expects and what kernel lockstat provides. 1. Right now the reports are produced only for spinlocks and mutexes. Userland lockstat does know about sx locks and has incorrect expectations about rw locks. 2. Kernel lockstat does not cover rm locks and lockmgr locks. 3. Spinning statistic is provided in units of spin loop count. In illumos it seems to be provided in nanoseconds just like for blocking. 4. lockstat(1) manual pages is not installed. I guess that this is because it is not in compliance with FreeBSD manual page formatting. Some thoughts. First, regarding #3. It is certainly hard comparing apples to oranges, especially when adaptive behavior is enabled, so there is some spinning and some blocking for the same locks. On the other hand, maintaining a loop count is very cheap. Always doing some timing would incur some overhead. Would it be worth while switching to nanoseconds report for spinning? Perhaps there is some way to apply the DTrace principle of not introducing overhead unless a probe is actually used to the lockstat probes? Currently, it is not so - the measurement are taken regardless of whether they will be going anywhere. Regarding #1. I came up with the following patch. It is WIP. If you'd have any improvements or suggestions for it, please share. Please note that some things like lock upgrades and downgrades are still not reported at all. commit d4bbb4e9a2a2fc071676ac356f1b7ab5678d1784 Author: Andriy Gapon Date: Sun Oct 20 13:16:44 2013 +0300 [wip] lockstat: teach about freebsd sx and rw locks Consider preserving original solaris/illumos code under ifdef. diff --git a/cddl/contrib/opensolaris/cmd/lockstat/lockstat.c b/cddl/contrib/opensolaris/cmd/lockstat/lockstat.c index 0a609d7..8b9e956 100644 --- a/cddl/contrib/opensolaris/cmd/lockstat/lockstat.c +++ b/cddl/contrib/opensolaris/cmd/lockstat/lockstat.c @@ -142,26 +142,30 @@ typedef struct ls_event_info { } ls_event_info_t; static ls_event_info_t g_event_info[LS_MAX_EVENTS] = { - { 'C', "Lock", "Adaptive mutex spin", "nsec", + { 'C', "Lock", "Adaptive mutex spin", "count", "lockstat:::adaptive-spin" }, { 'C', "Lock", "Adaptive mutex block", "nsec", "lockstat:::adaptive-block" }, - { 'C', "Lock", "Spin lock spin", "nsec", + { 'C', "Lock", "Spin lock spin", "count", "lockstat:::spin-spin" }, - { 'C', "Lock", "Thread lock spin", "nsec", + { 'C', "Lock", "Thread lock spin", "count", "lockstat:::thread-spin" }, - { 'C', "Lock", "R/W writer blocked by writer", "nsec", - "lockstat:::rw-block", "arg2 == 0 && arg3 == 1" }, - { 'C', "Lock", "R/W writer blocked by readers", "nsec", - "lockstat:::rw-block", "arg2 == 0 && arg3 == 0 && arg4" }, - { 'C', "Lock", "R/W reader blocked by writer", "nsec", - "lockstat:::rw-block", "arg2 != 0 && arg3 == 1" }, - { 'C', "Lock", "R/W reader blocked by write wanted", "nsec", - "lockstat:::rw-block", "arg2 != 0 && arg3 == 0 && arg4" }, - { 'C', "Lock", "Unknown event (type 8)", "units" }, - { 'C', "Lock", "Unknown event (type 9)", "units" }, - { 'C', "Lock", "Unknown event (type 10)", "units" }, - { 'C', "Lock", "Unknown event (type 11)", "units" }, + { 'C', "Lock", "R/W writer spin", "count", + "lockstat::rw_wlock:rw-spin" }, + { 'C', "Lock", "R/W writer block", "nsec", + "lockstat::rw_wlock:rw-block" }, + { 'C', "Lock", "R/W reader spin", "count", + "lockstat::rw_rlock:rw-spin" }, + { 'C', "Lock", "R/W reader block", "nsec", + "lockstat::rw_rlock:rw-block" }, + { 'C', "Lock", "SX shared spin", "count", + "lockstat::sx_slock:sx-spin" }, + { 'C', "Lock", "SX shared block", "nsec", + "lockstat::sx_slock:sx-block" }, + { 'C', "Lock", "SX exclsuive spin", "count", + "lockstat::sx_xlock:sx-spin" }, + { 'C', "Lock", "SX exclsuive block", "nsec", + "lockstat::sx_xlock:sx-block" }, { 'C', "Lock", "Unknown event (type 12)", "units" }, { 'C', "Lock", "Unknown event (type 13)", "units" }, { 'C', "Lock", "Unknown event (type 14)", "units" }, @@ -189,13 +193,17 @@ static ls_event_info_t g_event_info[LS_MAX_EVENTS] = { "lockstat:::spin-release", NULL, "lockstat:::spin-acquire" }, { 'H', "Lock", "R/W writer hold", "nsec", - "lockstat:::rw-release", "arg1 == 0", - "lockstat:::rw-acquire" }, + "lockstat::rw_wunlock:rw-release", NULL, + "lockstat::rw_wlock:rw-acquire" }, { 'H', "Lock", "R/W reader hold", "nsec", - "lockstat:::rw-release", "arg1 != 0", - "lockstat:::rw-acquire" }, - { 'H', "Lock", "Unknown event (type 36)", "units" }, - { 'H', "Lock", "Unknown event (type 37)", "units" }, + "lockstat::rw_runlock:rw-release", NULL, + "lockstat::rw_rlock:rw-acquire" }, + { 'H', "Lock", "SX shared hold", "nsec", + "lockstat::sx_sunlock:sx-release", NULL, + "lockstat::sx_slock:sx-acquire" }, + { 'H', "Lock", "SX exclusive hold", "nsec", + "lockstat::sx_xunlock:sx-release", NULL, + "lockstat::sx_xlock:sx-acquire" }, { 'H', "Lock", "Unknown event (type 38)", "units" }, { 'H', "Lock", "Unknown event (type 39)", "units" }, { 'H', "Lock", "Unknown event (type 40)", "units" }, -- Andriy Gapon From owner-freebsd-dtrace@FreeBSD.ORG Thu Oct 24 16:16:28 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D247E26E; Thu, 24 Oct 2013 16:16:28 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qc0-x235.google.com (mail-qc0-x235.google.com [IPv6:2607:f8b0:400d:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8326E28FF; Thu, 24 Oct 2013 16:16:28 +0000 (UTC) Received: by mail-qc0-f181.google.com with SMTP id w4so1496975qcr.12 for ; Thu, 24 Oct 2013 09:16:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=QmNXff5WehBM1ul37U8hPJjyOsVKXtiASBnRZL6biAA=; b=GfAgd9RDGTHx0Vfv4d/RFvIFnVoWf3aMmKrmteho9WpZy2T0dhapbYLIDOkxTIP+VQ B2WVSfxvXMwojp9G5aiQfODaeW+GesqcK2mQJqewhJi6Y5a8lAeGeUQEaQXKCogR+0/9 2SHuFqZ3sZ7NI2bHTP6q4n/AkKy5B4M8a+uVwbT1wBAvItJSpHZh3fEEcLxIQR1TjXQk 0UOTrXH4UWGopygUX0oeTxOulPbbeLkBdn7GsH693o9Oh/bbQzR/EhLFRhJwoIOrm99q Aw998XdFi4cmmzgXe+kjf2H08xTUIR1g8W5eheMj0uh0wUbQQGk0ckQScbUmGpXWs3yk /K1Q== X-Received: by 10.224.8.65 with SMTP id g1mr5175823qag.68.1382631387188; Thu, 24 Oct 2013 09:16:27 -0700 (PDT) Received: from charmander (mail1.sandvine.com. [64.7.137.162]) by mx.google.com with ESMTPSA id x10sm9066915qas.5.2013.10.24.09.16.25 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 24 Oct 2013 09:16:26 -0700 (PDT) Sender: Mark Johnston Date: Thu, 24 Oct 2013 12:16:20 -0400 From: Mark Johnston To: Andriy Gapon Subject: Re: "unstable" sdt probes Message-ID: <20131024161620.GA1710@charmander> References: <5268F461.7080504@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5268F461.7080504@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 16:16:28 -0000 On Thu, Oct 24, 2013 at 01:20:17PM +0300, Andriy Gapon wrote: > > Can our SDT (for kernel) implementation support probes with unspecified types > for arguments? > I would like to get DTRACE_PROBE*() which can be found in the code with > OpenSolaris origins (e.g. ZFS) to work. With minimal efforts :-) Hm, it looks like illumos uses the first argument to DTRACE_PROBE* to specify both the provider name and probe name; if no provider name is given (based on a lookup in a table in sdt_subr.c), a default "sdt" provider is used. It looks like this is what's happening for the ZFS probes. I'd suggest something like the following: - to kern_sdt.c, add SDT_PROVIDER_DEFINE(sdt); - add DTRACE_PROBE* macros to sdt.h which invoke SDT_PROBE* with the sdt provider, i.e. something like #define DTRACE_PROBE1(name, type, arg) \ SDT_PROBE1(sdt, , , name, arg) - add a FreeBSD-only zfs_dtrace.c which contains the SDT_PROBE_DEFINE* invocations for the ZFS probes. You can define the types there, or not (using an empty string or NULL should work, I'm not sure). This won't work for illumos code where the probes specify a provider, but I think that's ok. I can do the first two steps if you agree with this approach. I don't have any way to test the ZFS probes at the moment, but I guess I can provide a zfs_dtrace.c too if you (or anyone else) can test. Thanks, -Mark From owner-freebsd-dtrace@FreeBSD.ORG Fri Oct 25 10:47:14 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A60FB255 for ; Fri, 25 Oct 2013 10:47:14 +0000 (UTC) (envelope-from symbolics@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4F2BB2D39 for ; Fri, 25 Oct 2013 10:47:14 +0000 (UTC) Received: from lemon ([80.7.17.14]) by mail.gmx.com (mrgmx002) with ESMTPSA (Nemesis) id 0MBnvD-1VRiYe1bAl-00Ao12 for ; Fri, 25 Oct 2013 12:47:07 +0200 Received: by lemon (Postfix, from userid 1001) id AB7E7EB2E3; Fri, 25 Oct 2013 11:47:06 +0100 (BST) Date: Fri, 25 Oct 2013 11:47:06 +0100 From: symbolics@gmx.com To: dtrace@freebsd.org Subject: Re: Firefox crash during dtrace attach under -CURRENT Message-ID: <20131025104706.GB1705@lemon> References: <20131023203009.GA92945@lemon> <20131024025902.GA2286@charmander> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131024025902.GA2286@charmander> X-Provags-ID: V03:K0:x+qVz0eDdalDomlH/mGIN4wNM2vTP8drKoouNOTjqqIiX6gm1IB Ad1AFQbmhIyZP2Y+A5TYy27dJUfhJUuXvseHeuU8T2V+mZ7r12agSyaWkOec/vhpKeXIrMw QzNGI0pzbZmpdfVvBQkt8ce/k2D51sbmkX3qXVpo9RhvzwKH/Ds3Aibs9ECFHm/+2kjdvV+ hNCO+jFPNLGY3rk2an86w== X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 10:47:14 -0000 On Wed, Oct 23, 2013 at 10:59:02PM -0400, Mark Johnston wrote: > On Wed, Oct 23, 2013 at 09:30:09PM +0100, symbolics@gmx.com wrote: > > Hi, > > > > http://dtrace.org/blogs/brendan/2011/02/11/dtrace-pid-provider-arguments/ > > > > I tried to follow some of the examples but I crash the Firefox process > > each time. Sometimes DTrace manages to collect a little data before the > > death. > > > > [...] > > > > Is this a known problem or should I send a PR? > > Thanks for reporting this: I was able to reproduce the crash and managed > to find a nasty pair of bugs. Could you test the patch below and let me > know if it fixes the problem for you as well? If you see more crashes, > please include the backtrace and signo from gdb again; it would likely > be a different problem that needs to be debugged and fixed separately. Hi Mark, This helps but there still may be some issues. First time I used this I found that when I killed the DTrace process Firefox went down too with a SIGTRAP. I have a possibly unhelpful core from this: Loaded symbols for /libexec/ld-elf.so.1 #0 0x000000080061d541 in r_debug_state (rd=0x801354480, m=0x7ffffb394f98) at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 3491 { [New Thread 81b2da400 (LWP 100202/StreamTrans #12)] [New Thread 81c4fb000 (LWP 100200/DOM Worker)] [New Thread 81392b400 (LWP 100198/SSL Cert #2)] [New Thread 816c62400 (LWP 100197/DNS Resolver #5)] [New Thread 8160ccc00 (LWP 100196/Analysis Helper)] [New Thread 8160cd400 (LWP 100195/Analysis Helper)] [New Thread 8160cc400 (LWP 100194/Analysis Helper)] [New Thread 8176f2800 (LWP 100187/DNS Resolver #4)] [New Thread 8176f2400 (LWP 100186/DNS Resolver #3)] [New Thread 819db1000 (LWP 100185/mozStorage #7)] [New Thread 819dab000 (LWP 100183/mozStorage #6)] [New Thread 8199bf000 (LWP 100182/mozStorage #5)] [New Thread 8176e7c00 (LWP 100181/DNS Resolver #2)] [New Thread 8176e8000 (LWP 100180/URL Classifier)] [New Thread 8176ef000 (LWP 100179/mozStorage #4)] [New Thread 81392a800 (LWP 100178/SSL Cert #1)] [New Thread 818672400 (LWP 100177/Proxy Resolution)] [New Thread 818671000 (LWP 100176/Cert Verify)] [New Thread 8176e9c00 (LWP 100175/DNS Resolver #1)] [New Thread 818671c00 (LWP 100174/Image Scaler)] [New Thread 8176ed800 (LWP 100173/mozStorage #3)] [New Thread 815ccd800 (LWP 100172/mozStorage #2)] [New Thread 817051c00 (LWP 100171/mozStorage #1)] [New Thread 815ee4800 (LWP 100170/localStorage DB)] [New Thread 816c62000 (LWP 100167/ImageDecoder #1)] [New Thread 801a04000 (LWP 100166/HTML5 Parser)] [New Thread 8160c5c00 (LWP 100165/Cache I/O)] [New Thread 815133800 (LWP 100163/DOM Worker)] [New Thread 814eccc00 (LWP 100162/Timer)] [New Thread 813845c00 (LWP 100161/Hang Monitor)] [New Thread 813843400 (LWP 100160/JS Watchdog)] [New Thread 813842c00 (LWP 100159/firefox)] [New Thread 813842800 (LWP 100158/JS GC Helper)] [New Thread 813842400 (LWP 100157/Socket Thread)] [New Thread 801a09400 (LWP 100156/Gecko_IOThread)] [New Thread 801a02400 (LWP 100079/firefox)] (gdb) bt full #0 0x000000080061d541 in r_debug_state (rd=0x801354480, m=0x7ffffb394f98) at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 No locals. #1 0x000000000000000a in ?? () No symbol table info available. #2 0x00000001013f2260 in ?? () No symbol table info available. #3 0x00000008016b4a01 in _gmonparam () from /lib/libc.so.7 No symbol table info available. #4 0x000000021a067540 in ?? () No symbol table info available. #5 0x0000000000000020 in ?? () No symbol table info available. #6 0x0000000800000002 in ?? () No symbol table info available. #7 0x00007ffffb395000 in ?? () No symbol table info available. #8 0x00000008013e2cb4 in __jemalloc_tcache_alloc_small (tcache=0x800636000, size=, zero=133) at tcache.h:323 tbin = (tcache_bin_t *) 0x1a058b48197520a8 ret = Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal Another time I saw the DTrace process exit by itself. The Firefox process was still present but stuck in uwait with the UI completely unresponsive. I tried reattaching to it but that didn't go anywhere. Having said that, I can't seem to reproduce the crash consistently. I'll keep playing with this and see if I can break it. Finally, I still see the gelf_getehdr() error messages. If these aren't really errors, should they be announced? By the way, the DTrace command I used for testing was: pid$target::malloc:entry { @ = quantize(arg0); } profile-5sec { printa(@); } Thanks for your work! --sym From owner-freebsd-dtrace@FreeBSD.ORG Fri Oct 25 15:00:05 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 373CDEF4 for ; Fri, 25 Oct 2013 15:00:05 +0000 (UTC) (envelope-from symbolics@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C139C2BF3 for ; Fri, 25 Oct 2013 15:00:04 +0000 (UTC) Received: from lemon ([80.7.17.14]) by mail.gmx.com (mrgmx003) with ESMTPSA (Nemesis) id 0MILxX-1VYLgY1y0d-004AMF for ; Fri, 25 Oct 2013 16:59:57 +0200 Received: by lemon (Postfix, from userid 1001) id C3781EB3A6; Fri, 25 Oct 2013 15:59:56 +0100 (BST) Date: Fri, 25 Oct 2013 15:59:56 +0100 From: symbolics@gmx.com To: dtrace@freebsd.org Subject: Re: Firefox crash during dtrace attach under -CURRENT Message-ID: <20131025145956.GA26814@lemon> References: <20131023203009.GA92945@lemon> <20131024025902.GA2286@charmander> <20131025104706.GB1705@lemon> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131025104706.GB1705@lemon> X-Provags-ID: V03:K0:Gjk53Sz7YZ7wY6+GDeK06kqyYBQQ/PmkAx8c03oh1Trm/6FIB5S LOFpA7bF1Gi8g3sgwAzrbyZpqEunRwAlfeJfQpfTe4wHaLaWlMVa1qhRfYFAAgxa64fY0jH r7qegfS1XzZbJkbxVofPFR796Zy4255bX8sYBPlFK8Ww9WSbEixSGjaXre1iTTheLlrtrXE Dt3qgWbSr4UbqlG+Y9Ylg== X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 15:00:05 -0000 On Fri, Oct 25, 2013 at 11:47:06AM +0100, symbolics@gmx.com wrote: > On Wed, Oct 23, 2013 at 10:59:02PM -0400, Mark Johnston wrote: > > On Wed, Oct 23, 2013 at 09:30:09PM +0100, symbolics@gmx.com wrote: > > > Hi, > > > > > > http://dtrace.org/blogs/brendan/2011/02/11/dtrace-pid-provider-arguments/ > > > > > > I tried to follow some of the examples but I crash the Firefox process > > > each time. Sometimes DTrace manages to collect a little data before the > > > death. > > > > > > [...] > > > > > > Is this a known problem or should I send a PR? > > > > Thanks for reporting this: I was able to reproduce the crash and managed > > to find a nasty pair of bugs. Could you test the patch below and let me > > know if it fixes the problem for you as well? If you see more crashes, > > please include the backtrace and signo from gdb again; it would likely > > be a different problem that needs to be debugged and fixed separately. > > Hi Mark, > > This helps but there still may be some issues. First time I used this > I found that when I killed the DTrace process Firefox went down too > with a SIGTRAP. I have a possibly unhelpful core from this: > Another data point. I attached to mutt and reviewed some of the calls it was making. Subsequently I killed DTrace, went to to look at other things and a while later when back to check my mail. On attempting to change into a different mail folder mutt died with a SIGTRAP. It seems like DTrace isn't tidying up after itself? (gdb) bt #0 0x0000000800722541 in r_debug_state (rd=0x802425480, m=0x7fffffff6c28) at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 #1 0x0000000000000000 in ?? () --sym From owner-freebsd-dtrace@FreeBSD.ORG Fri Oct 25 15:09:29 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6636141E for ; Fri, 25 Oct 2013 15:09:29 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qc0-x22f.google.com (mail-qc0-x22f.google.com [IPv6:2607:f8b0:400d:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 25A602C91 for ; Fri, 25 Oct 2013 15:09:29 +0000 (UTC) Received: by mail-qc0-f175.google.com with SMTP id e16so2039428qcx.6 for ; Fri, 25 Oct 2013 08:09:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=Le6/XZh5IoLBo8Br8jnUb2iRcbgOhNydgBNUBsmDyUI=; b=Ns71h0wIappogdGNHCb2/MhRpKf+gQsi+Vyy5cxWqE6iaaZlJydVordRMKEKXjUdLA 0THCDBu+i/B0fokfoVBVZs7iCZoCuLP3EhvZEZtX/oQbRArbldUOzhCmOPLog94RiPi7 Wjuzk24Zfz/f68jNRbKPKtb4BpIcyhUw0WrNqt+qfgk7nxeLO1d+9pIalrTvSAgAAAJG jovVpCNJv7adUqLxfVkf8/hgoZpDR4oMF71eSlS2M4GcrsT9x7X/8PN6VSsE4Y+SHGHG Tve0nlVnr+OXnZXb7EACrnLL06x+d86EEpbOn2invdnlGmrVEmTgPlTsBmfPZ/KYvZ+s BxEQ== X-Received: by 10.229.73.6 with SMTP id o6mr11448605qcj.2.1382713768214; Fri, 25 Oct 2013 08:09:28 -0700 (PDT) Received: from charmander (mail1.sandvine.com. [64.7.137.162]) by mx.google.com with ESMTPSA id 5sm20246509qao.3.2013.10.25.08.09.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 25 Oct 2013 08:09:27 -0700 (PDT) Sender: Mark Johnston Date: Fri, 25 Oct 2013 11:09:23 -0400 From: Mark Johnston To: symbolics@gmx.com Subject: Re: Firefox crash during dtrace attach under -CURRENT Message-ID: <20131025150923.GA1906@charmander> References: <20131023203009.GA92945@lemon> <20131024025902.GA2286@charmander> <20131025104706.GB1705@lemon> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131025104706.GB1705@lemon> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 15:09:29 -0000 On Fri, Oct 25, 2013 at 11:47:06AM +0100, symbolics@gmx.com wrote: > On Wed, Oct 23, 2013 at 10:59:02PM -0400, Mark Johnston wrote: > > On Wed, Oct 23, 2013 at 09:30:09PM +0100, symbolics@gmx.com wrote: > > > Hi, > > > > > > http://dtrace.org/blogs/brendan/2011/02/11/dtrace-pid-provider-arguments/ > > > > > > I tried to follow some of the examples but I crash the Firefox process > > > each time. Sometimes DTrace manages to collect a little data before the > > > death. > > > > > > [...] > > > > > > Is this a known problem or should I send a PR? > > > > Thanks for reporting this: I was able to reproduce the crash and managed > > to find a nasty pair of bugs. Could you test the patch below and let me > > know if it fixes the problem for you as well? If you see more crashes, > > please include the backtrace and signo from gdb again; it would likely > > be a different problem that needs to be debugged and fixed separately. > > Hi Mark, > > This helps but there still may be some issues. First time I used this > I found that when I killed the DTrace process Firefox went down too > with a SIGTRAP. I have a possibly unhelpful core from this: > > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x000000080061d541 in r_debug_state (rd=0x801354480, m=0x7ffffb394f98) > at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 > 3491 { > [New Thread 81b2da400 (LWP 100202/StreamTrans #12)] > [New Thread 81c4fb000 (LWP 100200/DOM Worker)] > [New Thread 81392b400 (LWP 100198/SSL Cert #2)] > [New Thread 816c62400 (LWP 100197/DNS Resolver #5)] > [New Thread 8160ccc00 (LWP 100196/Analysis Helper)] > [New Thread 8160cd400 (LWP 100195/Analysis Helper)] > [New Thread 8160cc400 (LWP 100194/Analysis Helper)] > [New Thread 8176f2800 (LWP 100187/DNS Resolver #4)] > [New Thread 8176f2400 (LWP 100186/DNS Resolver #3)] > [New Thread 819db1000 (LWP 100185/mozStorage #7)] > [New Thread 819dab000 (LWP 100183/mozStorage #6)] > [New Thread 8199bf000 (LWP 100182/mozStorage #5)] > [New Thread 8176e7c00 (LWP 100181/DNS Resolver #2)] > [New Thread 8176e8000 (LWP 100180/URL Classifier)] > [New Thread 8176ef000 (LWP 100179/mozStorage #4)] > [New Thread 81392a800 (LWP 100178/SSL Cert #1)] > [New Thread 818672400 (LWP 100177/Proxy Resolution)] > [New Thread 818671000 (LWP 100176/Cert Verify)] > [New Thread 8176e9c00 (LWP 100175/DNS Resolver #1)] > [New Thread 818671c00 (LWP 100174/Image Scaler)] > [New Thread 8176ed800 (LWP 100173/mozStorage #3)] > [New Thread 815ccd800 (LWP 100172/mozStorage #2)] > [New Thread 817051c00 (LWP 100171/mozStorage #1)] > [New Thread 815ee4800 (LWP 100170/localStorage DB)] > [New Thread 816c62000 (LWP 100167/ImageDecoder #1)] > [New Thread 801a04000 (LWP 100166/HTML5 Parser)] > [New Thread 8160c5c00 (LWP 100165/Cache I/O)] > [New Thread 815133800 (LWP 100163/DOM Worker)] > [New Thread 814eccc00 (LWP 100162/Timer)] > [New Thread 813845c00 (LWP 100161/Hang Monitor)] > [New Thread 813843400 (LWP 100160/JS Watchdog)] > [New Thread 813842c00 (LWP 100159/firefox)] > [New Thread 813842800 (LWP 100158/JS GC Helper)] > [New Thread 813842400 (LWP 100157/Socket Thread)] > [New Thread 801a09400 (LWP 100156/Gecko_IOThread)] > [New Thread 801a02400 (LWP 100079/firefox)] > (gdb) bt full > #0 0x000000080061d541 in r_debug_state (rd=0x801354480, m=0x7ffffb394f98) > at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 > This is a function in rtld that's intended specifically for setting breakpoints prior to running init functions and main(); it's used by dtrace(1) too. I haven't seen this crash myself, but I'll see if I can trigger it once I have some time to work on this a bit more. > [...] > > Another time I saw the DTrace process exit by itself. The Firefox > process was still present but stuck in uwait with the UI completely > unresponsive. I tried reattaching to it but that didn't go anywhere. It seems I can reproduce this reliably by opening facebook while dtrace is attached. I'll try and figure out what's going on, thanks. > > Having said that, I can't seem to reproduce the crash consistently. I'll > keep playing with this and see if I can break it. Finally, I still see > the gelf_getehdr() error messages. If these aren't really errors, should > they be announced? Probably not. :) > > By the way, the DTrace command I used for testing was: > > pid$target::malloc:entry { @ = quantize(arg0); } > profile-5sec { printa(@); } > > Thanks for your work! No problem. Thanks for reporting these issues. -Mark From owner-freebsd-dtrace@FreeBSD.ORG Fri Oct 25 16:31:50 2013 Return-Path: Delivered-To: dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 78719BE8; Fri, 25 Oct 2013 16:31:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7ED84223C; Fri, 25 Oct 2013 16:31:49 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA17364; Fri, 25 Oct 2013 19:31:42 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1VZkIX-000Aeh-SG; Fri, 25 Oct 2013 19:31:42 +0300 Message-ID: <526A9CB5.2050207@FreeBSD.org> Date: Fri, 25 Oct 2013 19:30:45 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: Mark Johnston Subject: Re: "unstable" sdt probes References: <5268F461.7080504@FreeBSD.org> <20131024161620.GA1710@charmander> In-Reply-To: <20131024161620.GA1710@charmander> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 16:31:50 -0000 on 24/10/2013 19:16 Mark Johnston said the following: > On Thu, Oct 24, 2013 at 01:20:17PM +0300, Andriy Gapon wrote: >> >> Can our SDT (for kernel) implementation support probes with unspecified types >> for arguments? >> I would like to get DTRACE_PROBE*() which can be found in the code with >> OpenSolaris origins (e.g. ZFS) to work. With minimal efforts :-) > > Hm, it looks like illumos uses the first argument to DTRACE_PROBE* to > specify both the provider name and probe name; if no provider name is > given (based on a lookup in a table in sdt_subr.c), a default "sdt" > provider is used. It looks like this is what's happening for the ZFS > probes. > > I'd suggest something like the following: > - to kern_sdt.c, add > > SDT_PROVIDER_DEFINE(sdt); > > - add DTRACE_PROBE* macros to sdt.h which invoke SDT_PROBE* with the sdt > provider, i.e. something like > > #define DTRACE_PROBE1(name, type, arg) \ > SDT_PROBE1(sdt, , , name, arg) > > - add a FreeBSD-only zfs_dtrace.c which contains the SDT_PROBE_DEFINE* > invocations for the ZFS probes. You can define the types there, or not > (using an empty string or NULL should work, I'm not sure). > > This won't work for illumos code where the probes specify a provider, > but I think that's ok. I can do the first two steps if you agree with > this approach. I don't have any way to test the ZFS probes at the > moment, but I guess I can provide a zfs_dtrace.c too if you (or anyone > else) can test. Mark, thank you for the ideas! The approach sounds fine to me. I plan to have some time to work on this next week. I will definitely be able to test things and maybe even develop something. So, thank you again. BTW, I've been pondering an idea of reimplementing how the SDT probes get called. In FreeBSD we have a special hook function pointer that we check for not being NULL and then make a function call. In illumos they compile the code with an unconditional function call. This way the probe parameters are placed into the proper registers (or stack locations). But during run-time linking the call instructions are replaced with series of 1-byte NOP instructions (5 x 0x90 for amd64). When a probe gets activated then the first of those NOPs gets replaced with 0xf0 (lock prefix), which results in an invalid instruction (and that happens atomically). So, that allows for the SDT hook to be invoked via the trap handler. So, I think that that results in less overhead for inactive probes, but probably in more overhead for active probes. There is a trade off, but I believe that less overhead for inactive probes is preferred. -- Andriy Gapon From owner-freebsd-dtrace@FreeBSD.ORG Fri Oct 25 17:51:53 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0425CF46; Fri, 25 Oct 2013 17:51:53 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qe0-x233.google.com (mail-qe0-x233.google.com [IPv6:2607:f8b0:400d:c02::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A7F2026DC; Fri, 25 Oct 2013 17:51:52 +0000 (UTC) Received: by mail-qe0-f51.google.com with SMTP id q19so2574012qeb.38 for ; Fri, 25 Oct 2013 10:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=AsK/EIVvKCUQj3vmHSXtTrbgtl4kZlX5CRxloeWGhag=; b=AczjwDZNk1bTCJ4bO5HKRSJz/tvYraFVFf9oruXEOVkIsEO/byhq2oY/EXipGoMeVj inbVQABFA420mN7ieeNCFnPzzo/92ZD2/kFxg+h86qJYcdIm721pCSl0hRFBdl2nATab ziGcbaasB6X8vPJvLUA2CHorvIBZUKWUXhT0LKt0e5QngiGmF6SknexC4X9mMgdVcioN okzadhT8gGqJHf1E64vW/tBICa+LH3tzMGvnjcTNxHwTPEDv6D6iw2+4/bnZ9n4bBlsr titWJ/vwQCX5Vow0PaiRCJXYMJ3z6SH2pi7a9JEwhlQA3AF/GiKfkyyF+WLP0xCIqYUk 0YUQ== X-Received: by 10.224.44.1 with SMTP id y1mr12991370qae.35.1382723511711; Fri, 25 Oct 2013 10:51:51 -0700 (PDT) Received: from charmander (mail1.sandvine.com. [64.7.137.162]) by mx.google.com with ESMTPSA id fx6sm15440220qeb.1.2013.10.25.10.51.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 25 Oct 2013 10:51:49 -0700 (PDT) Sender: Mark Johnston Date: Fri, 25 Oct 2013 13:51:47 -0400 From: Mark Johnston To: Andriy Gapon Subject: Re: "unstable" sdt probes Message-ID: <20131025175147.GB1906@charmander> References: <5268F461.7080504@FreeBSD.org> <20131024161620.GA1710@charmander> <526A9CB5.2050207@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <526A9CB5.2050207@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 17:51:53 -0000 On Fri, Oct 25, 2013 at 07:30:45PM +0300, Andriy Gapon wrote: > on 24/10/2013 19:16 Mark Johnston said the following: > > On Thu, Oct 24, 2013 at 01:20:17PM +0300, Andriy Gapon wrote: > >> > >> Can our SDT (for kernel) implementation support probes with unspecified types > >> for arguments? > >> I would like to get DTRACE_PROBE*() which can be found in the code with > >> OpenSolaris origins (e.g. ZFS) to work. With minimal efforts :-) > > > > Hm, it looks like illumos uses the first argument to DTRACE_PROBE* to > > specify both the provider name and probe name; if no provider name is > > given (based on a lookup in a table in sdt_subr.c), a default "sdt" > > provider is used. It looks like this is what's happening for the ZFS > > probes. > > > > I'd suggest something like the following: > > - to kern_sdt.c, add > > > > SDT_PROVIDER_DEFINE(sdt); > > > > - add DTRACE_PROBE* macros to sdt.h which invoke SDT_PROBE* with the sdt > > provider, i.e. something like > > > > #define DTRACE_PROBE1(name, type, arg) \ > > SDT_PROBE1(sdt, , , name, arg) > > > > - add a FreeBSD-only zfs_dtrace.c which contains the SDT_PROBE_DEFINE* > > invocations for the ZFS probes. You can define the types there, or not > > (using an empty string or NULL should work, I'm not sure). > > > > This won't work for illumos code where the probes specify a provider, > > but I think that's ok. I can do the first two steps if you agree with > > this approach. I don't have any way to test the ZFS probes at the > > moment, but I guess I can provide a zfs_dtrace.c too if you (or anyone > > else) can test. > > Mark, > > thank you for the ideas! The approach sounds fine to me. > I plan to have some time to work on this next week. > I will definitely be able to test things and maybe even develop something. > So, thank you again. Ok, I'll try to do some of that stuff this weekend. :) > > BTW, I've been pondering an idea of reimplementing how the SDT probes get > called. In FreeBSD we have a special hook function pointer that we check for > not being NULL and then make a function call. > In illumos they compile the code with an unconditional function call. This way > the probe parameters are placed into the proper registers (or stack locations). > But during run-time linking the call instructions are replaced with series of > 1-byte NOP instructions (5 x 0x90 for amd64). When a probe gets activated then > the first of those NOPs gets replaced with 0xf0 (lock prefix), which results in > an invalid instruction (and that happens atomically). So, that allows for the > SDT hook to be invoked via the trap handler. > > So, I think that that results in less overhead for inactive probes, but probably > in more overhead for active probes. There is a trade off, but I believe that > less overhead for inactive probes is preferred. I'd like to find a good way of quantifying the overhead of the current approach when probes are disabled. Do you have any suggestions? One thing I'd like to try is just doing a TCP bulk transfer to localhost, since that'll trip the ip and tcp probes for each packet, and then just measure throughput with the current implementation and with the approach used in illumos. From owner-freebsd-dtrace@FreeBSD.ORG Fri Oct 25 18:05:21 2013 Return-Path: Delivered-To: dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CF6A338C; Fri, 25 Oct 2013 18:05:21 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9D7FD276D; Fri, 25 Oct 2013 18:05:21 +0000 (UTC) Received: from [209.249.190.124] (port=49418 helo=gnnmac.hudson-trading.com) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1) (envelope-from ) id 1VZll8-0007c6-1W; Fri, 25 Oct 2013 14:05:18 -0400 Content-Type: multipart/signed; boundary="Apple-Mail=_5ED9664E-A781-43F5-9A39-E92595CF34BF"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1816\)) Subject: Re: "unstable" sdt probes From: George Neville-Neil In-Reply-To: <20131025175147.GB1906@charmander> Date: Fri, 25 Oct 2013 14:05:20 -0400 Message-Id: <4B39604C-C678-43A1-9EFD-D1D7DE69CF02@neville-neil.com> References: <5268F461.7080504@FreeBSD.org> <20131024161620.GA1710@charmander> <526A9CB5.2050207@FreeBSD.org> <20131025175147.GB1906@charmander> To: Mark Johnston X-Mailer: Apple Mail (2.1816) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Cc: dtrace@FreeBSD.org, Andriy Gapon X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 18:05:22 -0000 --Apple-Mail=_5ED9664E-A781-43F5-9A39-E92595CF34BF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Oct 25, 2013, at 13:51 , Mark Johnston wrote: >>=20 >> BTW, I've been pondering an idea of reimplementing how the SDT probes = get >> called. In FreeBSD we have a special hook function pointer that we = check for >> not being NULL and then make a function call. >> In illumos they compile the code with an unconditional function call. = This way >> the probe parameters are placed into the proper registers (or stack = locations). >> But during run-time linking the call instructions are replaced with = series of >> 1-byte NOP instructions (5 x 0x90 for amd64). When a probe gets = activated then >> the first of those NOPs gets replaced with 0xf0 (lock prefix), which = results in >> an invalid instruction (and that happens atomically). So, that = allows for the >> SDT hook to be invoked via the trap handler. >>=20 >> So, I think that that results in less overhead for inactive probes, = but probably >> in more overhead for active probes. There is a trade off, but I = believe that >> less overhead for inactive probes is preferred. >=20 > I'd like to find a good way of quantifying the overhead of the current > approach when probes are disabled. Do you have any suggestions? One > thing I'd like to try is just doing a TCP bulk transfer to localhost, > since that'll trip the ip and tcp probes for each packet, and then > just measure throughput with the current implementation and with the > approach used in illumos. > _ Are you wanting to quantify just the workload involved with the probes = being on and off? If so, you want something simpler, and lower overhead than a TCP connection. Perhaps measuring fork() (not exec) with unixbench. I gather you=92re going to do this with hwpmc? Best, George --Apple-Mail=_5ED9664E-A781-43F5-9A39-E92595CF34BF Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlJqsuAACgkQYdh2wUQKM9IEoACgnaTjSwigUa4G/kJ34kiIaQji Y/oAoJe6r1LS/Mg92LKy6Im1FQMHBNiH =QYel -----END PGP SIGNATURE----- --Apple-Mail=_5ED9664E-A781-43F5-9A39-E92595CF34BF-- From owner-freebsd-dtrace@FreeBSD.ORG Fri Oct 25 18:21:29 2013 Return-Path: Delivered-To: dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C5C5F6BE; Fri, 25 Oct 2013 18:21:29 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CDAF72870; Fri, 25 Oct 2013 18:21:28 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id VAA18097; Fri, 25 Oct 2013 21:21:24 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1VZm0i-000Ali-IT; Fri, 25 Oct 2013 21:21:24 +0300 Message-ID: <526AB66C.6050106@FreeBSD.org> Date: Fri, 25 Oct 2013 21:20:28 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: George Neville-Neil , Mark Johnston Subject: Re: "unstable" sdt probes References: <5268F461.7080504@FreeBSD.org> <20131024161620.GA1710@charmander> <526A9CB5.2050207@FreeBSD.org> <20131025175147.GB1906@charmander> <4B39604C-C678-43A1-9EFD-D1D7DE69CF02@neville-neil.com> In-Reply-To: <4B39604C-C678-43A1-9EFD-D1D7DE69CF02@neville-neil.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 18:21:29 -0000 on 25/10/2013 21:05 George Neville-Neil said the following: > Are you wanting to quantify just the workload involved with the probes > being on and off? No. :-) Comparing different implementations of skipping off probes. -- Andriy Gapon From owner-freebsd-dtrace@FreeBSD.ORG Sat Oct 26 13:11:16 2013 Return-Path: Delivered-To: dtrace@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5508155D; Sat, 26 Oct 2013 13:11:16 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 248B22CD1; Sat, 26 Oct 2013 13:11:15 +0000 (UTC) Received: from [64.197.173.10] (port=59283 helo=[10.3.10.28]) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.80.1) (envelope-from ) id 1Va3cx-0005yO-Vw; Sat, 26 Oct 2013 09:10:05 -0400 Content-Type: multipart/signed; boundary="Apple-Mail=_7C3A0A85-A63C-42F2-9968-D6BEA3A3E50C"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1816\)) Subject: Re: "unstable" sdt probes From: George Neville-Neil In-Reply-To: <526AB66C.6050106@FreeBSD.org> Date: Sat, 26 Oct 2013 09:10:04 -0400 Message-Id: <224E501A-575D-4D17-A096-AF52F223413B@neville-neil.com> References: <5268F461.7080504@FreeBSD.org> <20131024161620.GA1710@charmander> <526A9CB5.2050207@FreeBSD.org> <20131025175147.GB1906@charmander> <4B39604C-C678-43A1-9EFD-D1D7DE69CF02@neville-neil.com> <526AB66C.6050106@FreeBSD.org> To: Andriy Gapon X-Mailer: Apple Mail (2.1816) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Cc: dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Oct 2013 13:11:16 -0000 --Apple-Mail=_7C3A0A85-A63C-42F2-9968-D6BEA3A3E50C Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii On Oct 25, 2013, at 14:20 , Andriy Gapon wrote: > on 25/10/2013 21:05 George Neville-Neil said the following: >> Are you wanting to quantify just the workload involved with the probes >> being on and off? > > No. :-) > > Comparing different implementations of skipping off probes. OK. Best, George --Apple-Mail=_7C3A0A85-A63C-42F2-9968-D6BEA3A3E50C Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlJrvywACgkQYdh2wUQKM9JhzACgqC15H6LJVnmlKDRV6+QMsfGP 8AMAn22FiXrHlDMJmfQi52Kh2dI0A8XF =8sx0 -----END PGP SIGNATURE----- --Apple-Mail=_7C3A0A85-A63C-42F2-9968-D6BEA3A3E50C-- From owner-freebsd-dtrace@FreeBSD.ORG Sat Oct 26 18:06:50 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 506FEC3; Sat, 26 Oct 2013 18:06:50 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 15A472B2D; Sat, 26 Oct 2013 18:06:50 +0000 (UTC) Received: by mail-ie0-f174.google.com with SMTP id qd12so8562537ieb.5 for ; Sat, 26 Oct 2013 11:06:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=wvV055pKpvrfveGKxU7ZoU6+vIyVCVCxhWdWcdFCVe8=; b=h/K/BKC6HfG4G4Yz+vojE1ikxPxTbwtfzwUOi/ekZ6iek42SsJGUHn9iatiCB8oWR0 1pAaQnz/ZmOo4NWjxG05Wr+nf3yEdXanr1vMtO/zFnkF/XWEZqQ55LDH3+C3jbyAyIRj puhMcDCxMxS2KPHZPooDS5as/s8bDAPG8M34TqdyNEIvteK7I2ha+a0rKRHPpJEeyNxk Vj7w2D7Ka0Er+BBnprK7QDXiW2it9mMCk45caHtSZduJ8zSQjfsrJNxKJbSqZ8c7Zy49 aR7IY6WJP/vQtxmO6eV2oTni+4zIl08WAlZkTfXM771BQelj5U+OJ4206Q69nxsOvAaH 9W0w== X-Received: by 10.50.39.51 with SMTP id m19mr2885797igk.51.1382810809088; Sat, 26 Oct 2013 11:06:49 -0700 (PDT) Received: from raichu (24-212-218-13.cable.teksavvy.com. [24.212.218.13]) by mx.google.com with ESMTPSA id cy15sm9613430igc.1.2013.10.26.11.06.47 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 26 Oct 2013 11:06:48 -0700 (PDT) Sender: Mark Johnston Date: Sat, 26 Oct 2013 14:06:43 -0400 From: Mark Johnston To: Andriy Gapon Subject: Re: "unstable" sdt probes Message-ID: <20131026180643.GA98676@raichu> References: <5268F461.7080504@FreeBSD.org> <20131024161620.GA1710@charmander> <526A9CB5.2050207@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <526A9CB5.2050207@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Oct 2013 18:06:50 -0000 On Fri, Oct 25, 2013 at 07:30:45PM +0300, Andriy Gapon wrote: > on 24/10/2013 19:16 Mark Johnston said the following: > > On Thu, Oct 24, 2013 at 01:20:17PM +0300, Andriy Gapon wrote: > >> > >> Can our SDT (for kernel) implementation support probes with unspecified types > >> for arguments? > >> I would like to get DTRACE_PROBE*() which can be found in the code with > >> OpenSolaris origins (e.g. ZFS) to work. With minimal efforts :-) > > > > Hm, it looks like illumos uses the first argument to DTRACE_PROBE* to > > specify both the provider name and probe name; if no provider name is > > given (based on a lookup in a table in sdt_subr.c), a default "sdt" > > provider is used. It looks like this is what's happening for the ZFS > > probes. > > > > I'd suggest something like the following: > > - to kern_sdt.c, add > > > > SDT_PROVIDER_DEFINE(sdt); > > > > - add DTRACE_PROBE* macros to sdt.h which invoke SDT_PROBE* with the sdt > > provider, i.e. something like > > > > #define DTRACE_PROBE1(name, type, arg) \ > > SDT_PROBE1(sdt, , , name, arg) > > > > - add a FreeBSD-only zfs_dtrace.c which contains the SDT_PROBE_DEFINE* > > invocations for the ZFS probes. You can define the types there, or not > > (using an empty string or NULL should work, I'm not sure). > > > > This won't work for illumos code where the probes specify a provider, > > but I think that's ok. I can do the first two steps if you agree with > > this approach. I don't have any way to test the ZFS probes at the > > moment, but I guess I can provide a zfs_dtrace.c too if you (or anyone > > else) can test. > > Mark, > > thank you for the ideas! The approach sounds fine to me. > I plan to have some time to work on this next week. > I will definitely be able to test things and maybe even develop something. > So, thank you again. The patch here is what I had in mind: http://people.freebsd.org/~markj/patches/zfs_probes.diff I've only compile-tested it, but it should create illumos-compatible ZFS probes without changing any ZFS code, assuming I understand exactly how they're creating/naming probes. :) -Mark > > BTW, I've been pondering an idea of reimplementing how the SDT probes get > called. In FreeBSD we have a special hook function pointer that we check for > not being NULL and then make a function call. > In illumos they compile the code with an unconditional function call. This way > the probe parameters are placed into the proper registers (or stack locations). > But during run-time linking the call instructions are replaced with series of > 1-byte NOP instructions (5 x 0x90 for amd64). When a probe gets activated then > the first of those NOPs gets replaced with 0xf0 (lock prefix), which results in > an invalid instruction (and that happens atomically). So, that allows for the > SDT hook to be invoked via the trap handler. > > So, I think that that results in less overhead for inactive probes, but probably > in more overhead for active probes. There is a trade off, but I believe that > less overhead for inactive probes is preferred. > > -- > Andriy Gapon From owner-freebsd-dtrace@FreeBSD.ORG Sat Oct 26 19:24:19 2013 Return-Path: Delivered-To: dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 3D84C843 for ; Sat, 26 Oct 2013 19:24:19 +0000 (UTC) (envelope-from rm@joyent.com) Received: from mail-qc0-f175.google.com (mail-qc0-f175.google.com [209.85.216.175]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F0F4B2E75 for ; Sat, 26 Oct 2013 19:24:18 +0000 (UTC) Received: by mail-qc0-f175.google.com with SMTP id e16so2739898qcx.34 for ; Sat, 26 Oct 2013 12:24:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=EFdkSuPnxx6arFVd2vdW1XXSU5yFUiwxNlMromtCn/w=; b=A3i9vOYebEsfU8AXa5IRfbcYolu93LygHpQUYyqsoKA+cbEP4Ul009bbtpyW6nNEmZ XfBDbYs1Feef50qR9zGAF8B3vMMXfx5ZHZukUPI8abKvzRj0aDY/cSFsbktKZiK+RIHe 53+AyrHi2NLVIGpTU/Dzjs7UEe+p6Kj++8Rmxn9SYwxoeQrZXmUr0/ZLt5EdIqo/HP8I iArrdLTy53KWDfYNvmctnPJmgiAg2z4OTOkm9I5MZtzawwYRcWAYZWNXudmugUaUMpAJ 648bt6Qpm8/Ow21U8yzHdQd0bVQp8W5Wh7jPZpfz3rzISF4FuPYq3RMK3rAjBP7/8igR 8h9w== X-Gm-Message-State: ALoCoQlh6o1+itJh8VGXe7tdRtPILhSPOY9IVTb5h/8EdCDsVXtpuW82wvvSNobhvgGQ0U4dcCxR X-Received: by 10.49.58.225 with SMTP id u1mr18981076qeq.55.1382814112036; Sat, 26 Oct 2013 12:01:52 -0700 (PDT) Received: from [192.168.100.127] (mobile-166-186-168-220.mycingular.net. [166.186.168.220]) by mx.google.com with ESMTPSA id h9sm33581113qaq.9.2013.10.26.12.01.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 26 Oct 2013 12:01:51 -0700 (PDT) Message-ID: <526C119C.3030806@joyent.com> Date: Sat, 26 Oct 2013 15:01:48 -0400 From: Robert Mustacchi User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Thunderbird/24.0.1 MIME-Version: 1.0 To: Mark Johnston Subject: Re: "unstable" sdt probes References: <5268F461.7080504@FreeBSD.org> <20131024161620.GA1710@charmander> <526A9CB5.2050207@FreeBSD.org> <20131026180643.GA98676@raichu> In-Reply-To: <20131026180643.GA98676@raichu> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: dtrace@FreeBSD.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Oct 2013 19:24:19 -0000 On 10/26/13 14:06 , Mark Johnston wrote: > On Fri, Oct 25, 2013 at 07:30:45PM +0300, Andriy Gapon wrote: >> on 24/10/2013 19:16 Mark Johnston said the following: >>> On Thu, Oct 24, 2013 at 01:20:17PM +0300, Andriy Gapon wrote: >>>> >>>> Can our SDT (for kernel) implementation support probes with unspecified types >>>> for arguments? >>>> I would like to get DTRACE_PROBE*() which can be found in the code with >>>> OpenSolaris origins (e.g. ZFS) to work. With minimal efforts :-) >>> >>> Hm, it looks like illumos uses the first argument to DTRACE_PROBE* to >>> specify both the provider name and probe name; if no provider name is >>> given (based on a lookup in a table in sdt_subr.c), a default "sdt" >>> provider is used. It looks like this is what's happening for the ZFS >>> probes. >>> >>> I'd suggest something like the following: >>> - to kern_sdt.c, add >>> >>> SDT_PROVIDER_DEFINE(sdt); >>> >>> - add DTRACE_PROBE* macros to sdt.h which invoke SDT_PROBE* with the sdt >>> provider, i.e. something like >>> >>> #define DTRACE_PROBE1(name, type, arg) \ >>> SDT_PROBE1(sdt, , , name, arg) >>> >>> - add a FreeBSD-only zfs_dtrace.c which contains the SDT_PROBE_DEFINE* >>> invocations for the ZFS probes. You can define the types there, or not >>> (using an empty string or NULL should work, I'm not sure). >>> >>> This won't work for illumos code where the probes specify a provider, >>> but I think that's ok. I can do the first two steps if you agree with >>> this approach. I don't have any way to test the ZFS probes at the >>> moment, but I guess I can provide a zfs_dtrace.c too if you (or anyone >>> else) can test. >> >> Mark, >> >> thank you for the ideas! The approach sounds fine to me. >> I plan to have some time to work on this next week. >> I will definitely be able to test things and maybe even develop something. >> So, thank you again. > > The patch here is what I had in mind: > http://people.freebsd.org/~markj/patches/zfs_probes.diff > > I've only compile-tested it, but it should create illumos-compatible ZFS > probes without changing any ZFS code, assuming I understand exactly how > they're creating/naming probes. :) Hi Mark, Your reading of what's going on in sdt_subr.c is right. We're basically just using a prefix on the SDT probe name to effectively namespace the probe to a provider. And if it's currently an unknown provider then we just end up putting it into the sdt module. All those zfs probes are just in the SDT provider for now, we don't have a stable ZFS provider yet. From my reading of what you've done it looks like you've created compatibly named probes. Robert