Date: Thu, 10 Jan 2013 01:56:48 -0500 From: Alfred Perlstein <bright@mu.org> To: Jason Evans <jasone@freebsd.org> Cc: Daniel Eischen <deischen@freebsd.org>, hackers@freebsd.org Subject: Re: malloc+utrace, tracking memory leaks in a running program. Message-ID: <50EE6630.2010902@mu.org> In-Reply-To: <50EE6281.7030602@mu.org> References: <50D52B10.1060205@mu.org> <A0AD197D-B72D-4FF5-B9AF-5E4F2AAAA421@freebsd.org> <50EE6281.7030602@mu.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/10/13 1:41 AM, Alfred Perlstein wrote: > On 12/23/12 12:28 PM, Jason Evans wrote: >> On Dec 21, 2012, at 7:37 PM, Alfred Perlstein <bright@mu.org> wrote: >>> So the other day in an effort to debug a memory leak I decided to >>> take a look at malloc+utrace(2) and decided to make a tool to debug >>> where leaks are coming from. >>> >>> A few hours later I have: >>> 1) a new version of utrace(2) (utrace2(2)) that uses structured data >>> to prevent overloading of data. (utrace2.diff) >>> 2) changes to ktrace and kdump to decode the new format. (also in >>> utrace2.diff) >>> 3) changes to jemalloc to include the new format AND the function >>> caller so it's easy to get the source of the leaks. (also in >>> utrace2.diff) >>> 4) a program that can take a pipe of kdump(1) and figure out what >>> memory has leaked. (alloctrace.py) >>> 5) simple test program (test_utrace.c) >>> >>> […] >> Have you looked at the heap profiling functionality built into >> jemalloc? It's not currently enabled on FreeBSD, but as far as I >> know, the only issue keeping it from being useful is the absence of a >> Linux-compatible /proc/<pid>/maps (and the gperftools folks may >> already have a solution for that; I haven't looked). I think it >> makes more sense to get that sorted out than to develop a separate >> trace-based leak checker. The problem with tracing is that it >> doesn't scale beyond some relatively small number of allocator events. > > I have looked at some of this functionality (heap profiling) but alas > it is not implemented yet. In addition the dtrace work appears to be > quite away from a workable solution with too many performance > penalties until some serious hacking is done. > > I am just not sure how to proceed, on one hand I do not really have > the skill to fix the /proc/pid/maps problem, nor figure out how to get > dtrace into the system in any time frame that is reasonable. > > All a few of us need is the addition of the trace back into the > existing utrace framework. > >>> Is it time to start installing with some form of debug symbols? This >>> would help us also with dtrace. >> Re: debug symbols, frame pointers, etc. necessary to make userland >> dtrace work by default, IMO we should strongly prefer such defaults. >> It's more reasonable to expect people who need every last bit of >> performance to remove functionality than to expect people who want to >> figure out what the system is doing to figure out what functionality >> to turn on. >> > > This is very true. I'm going to continue to work towards this end > with a few people and get up to speed on it so that hopefully we can > get to this point hopefully in the next release cycle or two. > > If you have a few moments, can you have a look at the "utrace2" > branches here: > https://github.com/alfredperlstein/freebsd/tree/utrace2 > > This branch contains the addition of the utrace2 system call which is > needed to structure data via utrace(2). The point of this is to avoid > kdump(1) needing to discern type of ktrace records based on arbitrary > size or other parameters and introduces an extensible protocol for new > types of utrace data. > > The utrace2 branch here augments jemalloc to use utrace2 to pass the > old utrace records, but in addition to pass the return address along > with the type and size of the allocation: > https://github.com/alfredperlstein/jemalloc/tree/utrace2 > > -Alfred Jason, Here are more convenient links that give diffs against FreeBSD and jemalloc for the proposed changes: FreeBSD: https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2 jemalloc: https://github.com/alfredperlstein/jemalloc/compare/master...utrace2 -Alfred
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50EE6630.2010902>