Date: Thu, 10 Jan 2013 01:41:05 -0500 From: Alfred Perlstein <bright@mu.org> To: Jason Evans <jasone@freebsd.org> Cc: Daniel Eischen <deischen@freebsd.org>, hackers@freebsd.org Subject: Re: malloc+utrace, tracking memory leaks in a running program. Message-ID: <50EE6281.7030602@mu.org> In-Reply-To: <A0AD197D-B72D-4FF5-B9AF-5E4F2AAAA421@freebsd.org> References: <50D52B10.1060205@mu.org> <A0AD197D-B72D-4FF5-B9AF-5E4F2AAAA421@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/23/12 12:28 PM, Jason Evans wrote: > On Dec 21, 2012, at 7:37 PM, Alfred Perlstein <bright@mu.org> wrote: >> So the other day in an effort to debug a memory leak I decided to take a look at malloc+utrace(2) and decided to make a tool to debug where leaks are coming from. >> >> A few hours later I have: >> 1) a new version of utrace(2) (utrace2(2)) that uses structured data to prevent overloading of data. (utrace2.diff) >> 2) changes to ktrace and kdump to decode the new format. (also in utrace2.diff) >> 3) changes to jemalloc to include the new format AND the function caller so it's easy to get the source of the leaks. (also in utrace2.diff) >> 4) a program that can take a pipe of kdump(1) and figure out what memory has leaked. (alloctrace.py) >> 5) simple test program (test_utrace.c) >> >> […] > Have you looked at the heap profiling functionality built into jemalloc? It's not currently enabled on FreeBSD, but as far as I know, the only issue keeping it from being useful is the absence of a Linux-compatible /proc/<pid>/maps (and the gperftools folks may already have a solution for that; I haven't looked). I think it makes more sense to get that sorted out than to develop a separate trace-based leak checker. The problem with tracing is that it doesn't scale beyond some relatively small number of allocator events. I have looked at some of this functionality (heap profiling) but alas it is not implemented yet. In addition the dtrace work appears to be quite away from a workable solution with too many performance penalties until some serious hacking is done. I am just not sure how to proceed, on one hand I do not really have the skill to fix the /proc/pid/maps problem, nor figure out how to get dtrace into the system in any time frame that is reasonable. All a few of us need is the addition of the trace back into the existing utrace framework. >> Is it time to start installing with some form of debug symbols? This would help us also with dtrace. > Re: debug symbols, frame pointers, etc. necessary to make userland dtrace work by default, IMO we should strongly prefer such defaults. It's more reasonable to expect people who need every last bit of performance to remove functionality than to expect people who want to figure out what the system is doing to figure out what functionality to turn on. > This is very true. I'm going to continue to work towards this end with a few people and get up to speed on it so that hopefully we can get to this point hopefully in the next release cycle or two. If you have a few moments, can you have a look at the "utrace2" branches here: https://github.com/alfredperlstein/freebsd/tree/utrace2 This branch contains the addition of the utrace2 system call which is needed to structure data via utrace(2). The point of this is to avoid kdump(1) needing to discern type of ktrace records based on arbitrary size or other parameters and introduces an extensible protocol for new types of utrace data. The utrace2 branch here augments jemalloc to use utrace2 to pass the old utrace records, but in addition to pass the return address along with the type and size of the allocation: https://github.com/alfredperlstein/jemalloc/tree/utrace2 -Alfred
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50EE6281.7030602>