From owner-freebsd-dtrace@FreeBSD.ORG Fri Dec 5 02:34:46 2014 Return-Path: Delivered-To: freebsd-dtrace@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B7A9A940; Fri, 5 Dec 2014 02:34:46 +0000 (UTC) Received: from mail-pa0-x22f.google.com (mail-pa0-x22f.google.com [IPv6:2607:f8b0:400e:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 80113E11; Fri, 5 Dec 2014 02:34:46 +0000 (UTC) Received: by mail-pa0-f47.google.com with SMTP id kq14so19245320pab.34 for ; Thu, 04 Dec 2014 18:34:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=xCyhdhg/Qfchee/SLUhO+QP+2Ysu/T7TDKxXsBoq7sU=; b=if+lo0+4K6Ulw7vr0eCsBm0S4Y3pHNjmJcl7eOZSYDJvXQFuipk+Q3ir7zMUAv7mGe X937PBCiSx0f+x9ocOnigD8gcXTipXMT+x5bjIQVn60ezKf+6jOqRn4epVhiIq63GTbY S45/2SZ2okTP3jQwM590+poyh5FoKSIKwcyE4WHcMJG7vMdSbF34Sm7uXLkPqgMNT22Y tqnI8M2TjS7yrb/SdKZUmoXuntX9s6RC1Jxz1bQO1RyEpBYlW6DkNTHrLwmk1d5eOQQi +ydunhvPxvXAYHJsWO4yj7WByq6YjEhozTCsGw0v5+qbc0JcoW49x7Xm/53dPuT7nqeO BZiQ== X-Received: by 10.68.69.80 with SMTP id c16mr31116833pbu.125.1417746885962; Thu, 04 Dec 2014 18:34:45 -0800 (PST) Received: from charmander.picturesperfect.net (c-67-182-131-225.hsd1.wa.comcast.net. [67.182.131.225]) by mx.google.com with ESMTPSA id kb16sm27241131pbb.34.2014.12.04.18.34.44 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 04 Dec 2014 18:34:45 -0800 (PST) Sender: Mark Johnston Date: Thu, 4 Dec 2014 18:34:39 -0800 From: 'Mark Johnston' To: dteske@FreeBSD.org Subject: Re: DTrace script to trace processes entering vfs::vop_remove Message-ID: <20141205023439.GA8710@charmander.picturesperfect.net> References: <032e01d00f4f$98a04e20$c9e0ea60$@FreeBSD.org> <20141204004501.GB29167@charmander.picturesperfect.net> <03ed01d00f66$89db0ee0$9d912ca0$@FreeBSD.org> <20141204182232.GB81713@charmander.picturesperfect.net> <04f001d01004$abaefd30$030cf790$@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <04f001d01004$abaefd30$030cf790$@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: 'Julian Elischer' , freebsd-dtrace@freebsd.org X-BeenThere: freebsd-dtrace@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "A discussion list for developers working on DTrace in FreeBSD." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2014 02:34:46 -0000 On Thu, Dec 04, 2014 at 12:55:40PM -0800, dteske@FreeBSD.org wrote: > > > > -----Original Message----- > > From: Mark Johnston [mailto:markjdb@gmail.com] On Behalf Of 'Mark > > Johnston' > > Sent: Thursday, December 4, 2014 10:23 AM > > To: dteske@FreeBSD.org > > Cc: freebsd-dtrace@freebsd.org; 'Julian Elischer' > > Subject: Re: DTrace script to trace processes entering vfs::vop_remove > > > > On Wed, Dec 03, 2014 at 06:03:45PM -0800, dteske@FreeBSD.org wrote: > > > > > > > > > > -----Original Message----- > > > > From: Mark Johnston [mailto:markjdb@gmail.com] On Behalf Of Mark > > > > Johnston > > > > Sent: Wednesday, December 3, 2014 4:45 PM > > > > To: dteske@FreeBSD.org > > > > Cc: freebsd-dtrace@freebsd.org; 'Julian Elischer' > > > > Subject: Re: DTrace script to trace processes entering vfs::vop_remove > > > > > > > > On Wed, Dec 03, 2014 at 03:19:31PM -0800, dteske@FreeBSD.org wrote: > > > > > Hi markj, list, > > > > > > > > > > I wrote a script for $work to help me find out "who on Earth > > > > > keeps deleting files XYZ?" from a particular storage server. > > > > > > > > > > Please find attached a copy of watch_vop_remove.d which > > > > > has the following sample output: > > > > > > > > > > 2014 Dec 3 11:58:52 rm[75596]: /tmp/foo > > > > > -+= 72846 0.0 -bash > > > > > \-+= 75589 0.0 /bin/bash /usr/home/support/bash_script > > > > > \-+= 75596 0.0 rm -f /tmp/foo > > > > > > > > > > The above sample output was displayed when executing the following > > shell > > > > > script: > > > > > > > > > > #!/bin/bash > > > > > touch /tmp/foo > > > > > rm -f /tmp/foo > > > > > > > > > > The output format displayed for each vop_remove() call is as > follows: > > > > > > > > > > DATE process[PID]: PATH_TO_DELETE > > > > > -+= GPID UID.GID grandparent_process [arguments (up to 3)] > > > > > \-+= PPID UID.GID parent_process [arguments (up to 3)] > > > > > \-+= PID UID.GID process [arguments (up to 3)] > > > > > > > > This is neat. I just had a few comments: > > > > - You can use walltimestamp when printing the date and time, instead > of > > > > timestamp + blah. > > > > > > I read that online as well, however: > > > walltimestamp appears to _always_ be zero. > > > > Right, it wasn't working properly on 8.0. :( > > > > gnn committed a fix for that as r238537. > > > > > > > > > > > > - It's possible to get the full argv of the current process with > > > > curpsinfo->pr_psargs. It can be done for other processes too; see > > > > /usr/lib/dtrace/psinfo.d. (This might not be true depending on the > > > > FreeBSD version you're on.) > > > > > > Thanks! I'll have a look. > > > > > > > - Running this script with a make -j4 buildkernel causes dtrace to run > > > > out of dynamic variable space. > > > > > > > > > > Any recommendation on how to fix that? > > > > > > #pragma D option dynvarsize=what_exactly? > > > (16m causes a warning that it's lowering the dynamic variable memory) > > > > It looks like a leak - once I start seeing the errors, no file removals > > are logged at all. Dynamic variables need to be set to 0 once they're > > finished with in order to release the consumed memory. > > > > Thanks! Should be fixed in the latest (attached) version > (watch_vop_remove2.d). > > However, I read here: > http://wikis.oracle.com/display/DTrace/Variables > > Quote: Always assign zero to associative array elements that are no longer > in use. > > And I read some more about the different variable types in DTrace: > http://dtrace.org/blogs/brendan/2011/11/25/dtrace-variable-types/ > > It would appear that I've solved the issue by getting rid of associative > arrays. > > Can you give the latest (attached) a try? Yup, it appears to fix the issue for me. Thanks! -Mark