Date: Wed, 30 Oct 2013 09:19:26 -0400 From: Mark Johnston <markj@freebsd.org> To: symbolics@gmx.com Cc: dtrace@freebsd.org Subject: Re: Firefox crash during dtrace attach under -CURRENT Message-ID: <20131030131926.GA83433@raichu> In-Reply-To: <20131030123716.GA2037@lemon> References: <20131023203009.GA92945@lemon> <20131024025902.GA2286@charmander> <20131025104706.GB1705@lemon> <20131025145956.GA26814@lemon> <20131027195307.GA3206@charmander.uwaterloo.ca> <20131030081507.GA1674@lemon> <20131030123716.GA2037@lemon>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 30, 2013 at 12:37:16PM +0000, symbolics@gmx.com wrote: > On Wed, Oct 30, 2013 at 08:15:07AM +0000, symbolics@gmx.com wrote: > > On Sun, Oct 27, 2013 at 03:53:07PM -0400, Mark Johnston wrote: > > > On Fri, Oct 25, 2013 at 03:59:56PM +0100, symbolics@gmx.com wrote: > > > > On Fri, Oct 25, 2013 at 11:47:06AM +0100, symbolics@gmx.com wrote: > > > > > On Wed, Oct 23, 2013 at 10:59:02PM -0400, Mark Johnston wrote: > > > > > > On Wed, Oct 23, 2013 at 09:30:09PM +0100, symbolics@gmx.com wrote: > > > > > > > Hi, > > > > > > > > > > > > > > http://dtrace.org/blogs/brendan/2011/02/11/dtrace-pid-provider-arguments/ > > > > > > > > > > > > > > I tried to follow some of the examples but I crash the Firefox process > > > > > > > each time. Sometimes DTrace manages to collect a little data before the > > > > > > > death. > > > > > > > > > > > > > > [...] > > > > > > > > > > > > > > Is this a known problem or should I send a PR? > > > > > > > > > > > > Thanks for reporting this: I was able to reproduce the crash and managed > > > > > > to find a nasty pair of bugs. Could you test the patch below and let me > > > > > > know if it fixes the problem for you as well? If you see more crashes, > > > > > > please include the backtrace and signo from gdb again; it would likely > > > > > > be a different problem that needs to be debugged and fixed separately. > > > > > > > > > > Hi Mark, > > > > > > > > > > This helps but there still may be some issues. First time I used this > > > > > I found that when I killed the DTrace process Firefox went down too > > > > > with a SIGTRAP. I have a possibly unhelpful core from this: > > > > > > > > > > > > > Another data point. I attached to mutt and reviewed some of the calls it > > > > was making. Subsequently I killed DTrace, went to to look at other > > > > things and a while later when back to check my mail. On attempting to > > > > change into a different mail folder mutt died with a SIGTRAP. It seems > > > > like DTrace isn't tidying up after itself? > > > > > > > > (gdb) bt > > > > #0 0x0000000800722541 in r_debug_state (rd=0x802425480, m=0x7fffffff6c28) > > > > at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 > > > > #1 0x0000000000000000 in ?? () > > > > > > Ok, I think I've figured out this one too. As you note, dtrace(1) isn't > > > cleaning up some of its breakpoints properly when it detaches. In > > > particular, it's not stopping the victim process before it tries to > > > remove breakpoints using ptrace(2); however, ptrace requires the target > > > process to be stopped, else it will return EBUSY. So the breakpoint in > > > the rtld gets left behind, and it turns out that r_debug_state() is called > > > every time a process tries to dlopen() a shared object. > > > > > > mutt was a good example since it seems to dlopen() iconv-related stuff > > > as I scan through my inbox; one can inspect this with DTrace. :) > > > i.e. with something like > > > > > > 'pid$target::dlopen:entry {trace(copyinstr(arg0));}' > > > > > > With this observation it becomes easy to reproduce the problem using a > > > test program that does something like > > > > > > while (1) { > > > dlopen("/lib/libnonexistent.so.100", RTLD_LAZY); > > > sleep(1); > > > } > > > > > > A somewhat crude patch which fixes this for me is below; it just adds > > > code to send SIGSTOP to the target process before trying to remove > > > breakpoints. Does anyone see any problems with this? Perhaps it should > > > be libproc's responsibility to ensure that the victim process is stopped > > > before trying a ptrace(PT_IO, ...) to add/remove breakpoints? > > > > > > Thanks, > > > -Mark > > > > > > > Hi Mark, > > > > I've tried the patch but I can still reproduce the crash using mutt. I > > attached to the running mutt process with dtruss and watched that work a > > little bit, killed dtruss and carried on using mutt. I then tried to > > open a large mail folder and it crashed with a SIGTRAP. Backtrace, FWIW: > > > > (gdb) bt > > #0 0x0000000800722541 in r_debug_state (rd=0x802425480, m=0x7fffffff84a8) > > at /usr/home/dm/git/freebsd/libexec/rtld-elf/rtld.c:3491 > > #1 0x0000000000000001 in ?? () > > #2 0x0000000102c00000 in ?? () > > #3 0x0000000000000001 in ?? () > > #4 0x00000002028000c0 in ?? () > > #5 0x0000000000000001 in ?? () > > #6 0x0000000000000002 in ?? () > > #7 0x00007fffffff8d00 in ?? () > > #8 0x0000000801f25e3c in arena_avail_insert (arena=0xffffffff, > > chunk=0x7fffffff84a8, pageind=34395310320, npages=<value optimized out>, > > maybe_adjac_pred=<value optimized out>, maybe_adjac_succ=false) > > at /usr/obj/usr/home/dm/git/freebsd/lib/libc/jemalloc_arena.c:274 > > Previous frame inner to this frame (corrupt stack?) > > Current language: auto; currently minimal > > > > Just quickly adding, this crash & trace is perfectly reproducible. I did > a second round of kernel building and testing to make sure I hadn't > messed up. Ah, my apologies: I should have mentioned that this is not a kernel change but rather a change to libdtrace, though I will probably move this change to libproc instead when I have a chance. The easiest way to rebuild libdtrace, assuming that your sources are in sync with what's on the box, is to run # cd $SRCBASE/cddl/lib/libdtrace # make && make install Then dtrace(1) will be using the patched libdtrace.so. -Mark
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131030131926.GA83433>