From owner-freebsd-dtrace@FreeBSD.ORG Tue Feb 25 01:59:06 2014 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E4319294 for ; Tue, 25 Feb 2014 01:59:06 +0000 (UTC) Received: from mail-ie0-x231.google.com (mail-ie0-x231.google.com [IPv6:2607:f8b0:4001:c03::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AB66810DE for ; Tue, 25 Feb 2014 01:59:06 +0000 (UTC) Received: by mail-ie0-f177.google.com with SMTP id rp18so4081000iec.8 for ; Mon, 24 Feb 2014 17:59:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=r7yTU2vlqBFEZjPLgadh3QZYwVerCXz4A6z6s5K2fQw=; b=g4y1qxANPXZkb1rBJw74ePVgqr/itzkdeL2trDTNiP5riEMTnvlSt2I2kZFGsxWfvl NPTBE1huPce54P5X1NASrirK8xAGQ2XgMUJoDLCfTYd1NrenLriPbluVh6wzhb+6DA/G orzKBCnn2eFm8KWrljUQVWNI+bY8O4ac5kI2O2l11/YRwt99237uGbAC0r2TAHO7ae7r FmKRr+tFYvF5v5q1fMTmtjTXZKwYJvK6EoQVeMSRtuzMO8GqRtqKL2fUmWdfJNwhOGS8 9INrAzwGfmEh/12YW1voZYpErx1nL4gDLBrbJW4rTKGhJP9i69MiDYuF47paBLRtbpJL aXHA== X-Received: by 10.43.75.9 with SMTP id yy9mr17217628icb.54.1393293546166; Mon, 24 Feb 2014 17:59:06 -0800 (PST) Received: from raichu (198-84-185-216.cpe.teksavvy.com. [198.84.185.216]) by mx.google.com with ESMTPSA id ai4sm32518514igd.3.2014.02.24.17.59.04 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Feb 2014 17:59:05 -0800 (PST) Sender: Mark Johnston Date: Mon, 24 Feb 2014 20:59:03 -0500 From: Mark Johnston To: Adam Leventhal Subject: Re: [patch] fasttrap process scratch space Message-ID: <20140225015903.GB64934@raichu> References: <20140224041454.GB2720@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.22 (2013-10-16) Cc: freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Feb 2014 01:59:07 -0000 On Mon, Feb 24, 2014 at 09:51:13AM -0800, Adam Leventhal wrote: > Hey Mark, > > I wanted to provide a little historical perspective when considering > the options of a pact with libc/libthread/ld.so.1 vs. the kernel > mapping pages for TLS. > > We considered both but chose the former in Solaris. > Solaris/OpenSolaris/illumos have the kernel, libc, and ld.so.1 in the > same repository. There are already dependencies between those > components, so adding another was not a concern. That makes sense. Obviously, the same is true on FreeBSD, but what mostly worried me is the lack of any way to determine whether the traced process does in fact have scratch space available in its TLS. What if the executable is statically-linked with a libthr without my change? What if I'm attempting to trace a Linux binary running in the compatibility layer? In these cases, the program will just crash once it tries to execute instructions in non-executable memory (or it'll corrupt the thread control block), and I don't see any way to detect or prevent that in fasttrap. FreeBSD's DTrace implementation also tries to be somewhat compartmentalized so that it's possible to remove or add DTrace support without too much work. To my knowledge, it's all currently implemented using kernel modules and some userland executables and libraries. Requiring libthr and rtld support would take us in the opposite direction. > We felt that having > DTrace map pages into the traced process could have a more significant > impact on its execution, and we wanted to minimize the chance that > DTrace would chase away the very problem users were trying to > investigate. Further, we felt that the failure modes would be less > clean; for example, in your patch if we fail to map a page while in > fasttrap_pid_probe(), we're forced to silently remove the > instrumentation. That's true. It seemed to me that having to map 4 KB for every 64 threads in the process is not too much overhead, but it'd certainly be preferable to avoid it. In your much wider experience with userland DTrace, do you know of use cases where this might be likely to cause problems? > > Hope that's helpful. It is, I appreciate the explanation. I didn't mean to imply that the solution used in Solaris is inferior to the approch I followed; I just felt that it's not so well-suited to FreeBSD. Thanks! -Mark > > Adam > > On Sun, Feb 23, 2014 at 8:14 PM, Mark Johnston wrote: > > Hello, > > > > For those not familiar with MD parts of fasttrap, one of the things it > > has to do is ensure that any userland instruction that it replaces with > > a breakpoint gets executed in the traced process' context. For several > > common classes of instructions, fasttrap will emulate the instruction in > > the breakpoint handler; when it can't do that, it copies the instruction > > out to some scratch space in the process' address space and sets the PC > > of the interrupted thread to the address of that instruction, which is > > followed by a jump to the instruction following the breakpoint. There's > > a helpful block comment titled "Generic Instruction Tracing" around line > > 1585 of the x86 fasttrap_isa.c which describes the details of this. > > > > This functionality currently doesn't work on FreeBSD, mainly because we > > don't necessarily have any (per-thread) scratch space available for use > > in the process' address space. In illumos/Solaris, a small (< 64 byte) > > block is reserved in each thread's TLS for use by DTrace. It turns out > > that doing the same thing on FreeBSD is quite easy: > > > > http://people.freebsd.org/~markj/patches/fasttrap_scratch_hacky.diff > > > > Specifically, we need to ensure that TLS (allocated by the runtime > > linker) is executable and that we properly extract the offset to the > > scratch space from the FS segment register. I think this is somewhat > > hacky though, as it creates a dependency on libthr and rtld internals. > > > > A second approach is to have fasttrap dynamically allocate scratch space > > within the process' address space using vm_map_insert(9). My > > understanding is that Apple's DTrace implementation does this, and I've > > implemented this approach for FreeBSD here (which was done without > > referencing Apple code): > > > > http://people.freebsd.org/~markj/patches/fasttrap-scratch-space/fasttrap-scratch-space-1.diff > > > > The idea is to map pages of executable memory into the user process as > > needed, and carve them into scratch space chunks for use by individual > > threads. If a thread in fasttrap_pid_probe() needs scratch space, it > > calls a new function, fasttrap_scraddr(). If the thread already has > > scratch space allocated to it, it's used. Otherwise, if any free scratch > > space chunks are available in an already-mapped page, one of them is > > allocated to the thread and used. Otherwise, a new page is mapped using > > vm_map_insert(9). > > > > Threads hold onto their scratch space until they exit. That is, scratch > > space is never unmapped from the process, even if the controlling > > dtrace(1) process detaches. I added a handler for thread_dtor event > > which re-adds any scratch space held by the thread to the free list for > > that process. Per-process scratch space state is held in the fasttrap > > process handle (fasttrap_proc_t), since that turns out to be much easier > > than keeping it in the struct proc. > > > > Does anyone have any thoughts or comments on the approach or the patch? > > Any review or testing would be very much appreciated. > > > > For testing purposes, it's helpful to know that tracing memcpy() on > > amd64 will result in use of this scratch space code, as it starts with a > > "mov %rdi,%rax" on my machine at least. My main test case has been to > > run something like > > > > # dtrace -n 'pid$target:libc.so.7::entry {@[probefunc] = count()}' -p $(pgrep firefox) > > > > Attempting to trace all functions still results in firefox dying with > > SIGTRAP, but we're getting there. :) > > > > Thanks, > > -- > > -Mark > > _______________________________________________ > > freebsd-dtrace@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-dtrace > > To unsubscribe, send any mail to "freebsd-dtrace-unsubscribe@freebsd.org" > > > > -- > Adam Leventhal > CTO, Delphix > http://blog.delphix.com/ahl -- -Mark