From owner-freebsd-dtrace@FreeBSD.ORG Mon Feb 24 04:22:50 2014 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ED85E4F5 for ; Mon, 24 Feb 2014 04:22:50 +0000 (UTC) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B917E1D1B for ; Mon, 24 Feb 2014 04:22:50 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id rl12so3071903iec.40 for ; Sun, 23 Feb 2014 20:22:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=hSKgBnotm9G1cMtSpQE4ArlixGdU9yI0Pl6xGSnKvuM=; b=A6EwDIwhikBGqlYSg3FgCDyvFK0VClEBoDWfXDdxs3WrV+FYCeEq6mXpV0sXf8O3lY DdoXBXLTyLtdokWx6DlSkLFqNxLjBMgpblcQ5kMS0v+nyCLzf45GG73/UbRDKQP+S/ue F+dUCx4hs5ILd9HkDY311nIc1Ry+hixnII/qX0aBS21uzrIK4iQrb7XNldwfWp3Zotuw Y/LPu4clxb9HGmNfrue1PhIq375oPO0M/bbhvL99ZngjTr7CXucBgLZUqh2cHnaiThLr LrSBE7PgoN6Xos2KQ6vdlwnC9MicHi4VBr92c12OKGjR4P3t6J9LWTLxP1FoJM+WupwY Jv5A== X-Received: by 10.50.153.79 with SMTP id ve15mr12025890igb.40.1393215770210; Sun, 23 Feb 2014 20:22:50 -0800 (PST) Received: from raichu (198-84-185-216.cpe.teksavvy.com. [198.84.185.216]) by mx.google.com with ESMTPSA id dz8sm22680708igb.5.2014.02.23.20.22.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 23 Feb 2014 20:22:49 -0800 (PST) Sender: Mark Johnston Date: Sun, 23 Feb 2014 23:22:47 -0500 From: Mark Johnston To: freebsd-dtrace@freebsd.org Subject: Re: [patch] fasttrap process scratch space Message-ID: <20140224042247.GC2720@raichu> References: <20140224041454.GB2720@raichu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140224041454.GB2720@raichu> User-Agent: Mutt/1.5.22 (2013-10-16) X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Feb 2014 04:22:51 -0000 On Sun, Feb 23, 2014 at 11:14:54PM -0500, Mark Johnston wrote: > Hello, > > For those not familiar with MD parts of fasttrap, one of the things it > has to do is ensure that any userland instruction that it replaces with > a breakpoint gets executed in the traced process' context. For several > common classes of instructions, fasttrap will emulate the instruction in > the breakpoint handler; when it can't do that, it copies the instruction > out to some scratch space in the process' address space and sets the PC > of the interrupted thread to the address of that instruction, which is > followed by a jump to the instruction following the breakpoint. There's > a helpful block comment titled "Generic Instruction Tracing" around line > 1585 of the x86 fasttrap_isa.c which describes the details of this. > > This functionality currently doesn't work on FreeBSD, mainly because we > don't necessarily have any (per-thread) scratch space available for use > in the process' address space. In illumos/Solaris, a small (< 64 byte) > block is reserved in each thread's TLS for use by DTrace. It turns out > that doing the same thing on FreeBSD is quite easy: > > http://people.freebsd.org/~markj/patches/fasttrap_scratch_hacky.diff > > Specifically, we need to ensure that TLS (allocated by the runtime > linker) is executable and that we properly extract the offset to the > scratch space from the FS segment register. I think this is somewhat > hacky though, as it creates a dependency on libthr and rtld internals. > > A second approach is to have fasttrap dynamically allocate scratch space > within the process' address space using vm_map_insert(9). My > understanding is that Apple's DTrace implementation does this, and I've > implemented this approach for FreeBSD here (which was done without > referencing Apple code): > > http://people.freebsd.org/~markj/patches/fasttrap-scratch-space/fasttrap-scratch-space-1.diff > > The idea is to map pages of executable memory into the user process as > needed, and carve them into scratch space chunks for use by individual > threads. If a thread in fasttrap_pid_probe() needs scratch space, it > calls a new function, fasttrap_scraddr(). If the thread already has > scratch space allocated to it, it's used. Otherwise, if any free scratch > space chunks are available in an already-mapped page, one of them is > allocated to the thread and used. Otherwise, a new page is mapped using > vm_map_insert(9). > > Threads hold onto their scratch space until they exit. That is, scratch > space is never unmapped from the process, even if the controlling > dtrace(1) process detaches. I added a handler for thread_dtor event > which re-adds any scratch space held by the thread to the free list for > that process. Per-process scratch space state is held in the fasttrap > process handle (fasttrap_proc_t), since that turns out to be much easier > than keeping it in the struct proc. > > Does anyone have any thoughts or comments on the approach or the patch? > Any review or testing would be very much appreciated. > > For testing purposes, it's helpful to know that tracing memcpy() on > amd64 will result in use of this scratch space code, as it starts with a > "mov %rdi,%rax" on my machine at least. My main test case has been to > run something like > > # dtrace -n 'pid$target:libc.so.7::entry {@[probefunc] = count()}' -p $(pgrep firefox) > > Attempting to trace all functions still results in firefox dying with > SIGTRAP, but we're getting there. :) I should probably add that the diff described here should also be applied when testing: http://lists.freebsd.org/pipermail/freebsd-dtrace/2014-February/000175.html Otherwise it's quite easy to trigger deadlocks. -Mark