Date: Sun, 23 Dec 2012 10:02:19 -0800 From: Alfred Perlstein <bright@mu.org> To: Jason Evans <jasone@freebsd.org> Cc: Daniel Eischen <deischen@freebsd.org>, hackers@freebsd.org, Rui Paulo <rpaulo@freebsd.org> Subject: Re: malloc+utrace, tracking memory leaks in a running program. Message-ID: <50D7472B.4070604@mu.org> In-Reply-To: <A0AD197D-B72D-4FF5-B9AF-5E4F2AAAA421@freebsd.org> References: <50D52B10.1060205@mu.org> <A0AD197D-B72D-4FF5-B9AF-5E4F2AAAA421@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/23/12 9:28 AM, Jason Evans wrote: > On Dec 21, 2012, at 7:37 PM, Alfred Perlstein <bright@mu.org> wrote: >> So the other day in an effort to debug a memory leak I decided to take a look at malloc+utrace(2) and decided to make a tool to debug where leaks are coming from. >> >> A few hours later I have: >> 1) a new version of utrace(2) (utrace2(2)) that uses structured data to prevent overloading of data. (utrace2.diff) >> 2) changes to ktrace and kdump to decode the new format. (also in utrace2.diff) >> 3) changes to jemalloc to include the new format AND the function caller so it's easy to get the source of the leaks. (also in utrace2.diff) >> 4) a program that can take a pipe of kdump(1) and figure out what memory has leaked. (alloctrace.py) >> 5) simple test program (test_utrace.c) >> >> […] > Have you looked at the heap profiling functionality built into jemalloc? It's not currently enabled on FreeBSD, but as far as I know, the only issue keeping it from being useful is the absence of a Linux-compatible /proc/<pid>/maps (and the gperftools folks may already have a solution for that; I haven't looked). I think it makes more sense to get that sorted out than to develop a separate trace-based leak checker. The problem with tracing is that it doesn't scale beyond some relatively small number of allocator events. Ok, we are in agreement on this all. Paul Saab recommended profiling to me, but yes, the problem is that none of this stuff works on FreeBSD out of the box due to missing bits here or there. Augmenting the existing utrace stuff to get what I needed seemed much simpler than figuring out how to get dtrace, pidmaps and whatnot into the system. It's a matter of the requirements to accomplish these higher order things requires X=(skill+time+ability_to_socialize_these_changes) where X > alfred->skill_and_time_and_socialize(). :) To be honest, if dtrace just worked, then I could get the same information I'm getting from utrace2(2) from dtrace with no problem. (at least I think so). As far as scaling it, I agree it does not work for long running programs, however there are a few instances of programs leaking large memory in a short while that I can track down by temporarily ktracing for short while. >> Is it time to start installing with some form of debug symbols? This would help us also with dtrace. > Re: debug symbols, frame pointers, etc. necessary to make userland dtrace work by default, IMO we should strongly prefer such defaults. It's more reasonable to expect people who need every last bit of performance to remove functionality than to expect people who want to figure out what the system is doing to figure out what functionality to turn on. Yes!!! :) Is there an easy way to go about this? Rui says it's really a matter of just turning off stripping of shlibs and adding -fno-omit-frame-pointer and WITH_CTF. I'm going to give this a shot, if it works, can you help me refine this? I'll post diffs later today if I don't get completely stuck somehow. -Alfred
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50D7472B.4070604>