From owner-freebsd-hackers@FreeBSD.ORG Thu Jan 10 06:56:53 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 09A32DB6; Thu, 10 Jan 2013 06:56:53 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id D22766E9; Thu, 10 Jan 2013 06:56:52 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (unknown [207.238.187.242]) by elvis.mu.org (Postfix) with ESMTPSA id 770071A3D00; Wed, 9 Jan 2013 22:56:51 -0800 (PST) Message-ID: <50EE6630.2010902@mu.org> Date: Thu, 10 Jan 2013 01:56:48 -0500 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Jason Evans Subject: Re: malloc+utrace, tracking memory leaks in a running program. References: <50D52B10.1060205@mu.org> <50EE6281.7030602@mu.org> In-Reply-To: <50EE6281.7030602@mu.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: Daniel Eischen , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jan 2013 06:56:53 -0000 On 1/10/13 1:41 AM, Alfred Perlstein wrote: > On 12/23/12 12:28 PM, Jason Evans wrote: >> On Dec 21, 2012, at 7:37 PM, Alfred Perlstein wrote: >>> So the other day in an effort to debug a memory leak I decided to >>> take a look at malloc+utrace(2) and decided to make a tool to debug >>> where leaks are coming from. >>> >>> A few hours later I have: >>> 1) a new version of utrace(2) (utrace2(2)) that uses structured data >>> to prevent overloading of data. (utrace2.diff) >>> 2) changes to ktrace and kdump to decode the new format. (also in >>> utrace2.diff) >>> 3) changes to jemalloc to include the new format AND the function >>> caller so it's easy to get the source of the leaks. (also in >>> utrace2.diff) >>> 4) a program that can take a pipe of kdump(1) and figure out what >>> memory has leaked. (alloctrace.py) >>> 5) simple test program (test_utrace.c) >>> >>> […] >> Have you looked at the heap profiling functionality built into >> jemalloc? It's not currently enabled on FreeBSD, but as far as I >> know, the only issue keeping it from being useful is the absence of a >> Linux-compatible /proc//maps (and the gperftools folks may >> already have a solution for that; I haven't looked). I think it >> makes more sense to get that sorted out than to develop a separate >> trace-based leak checker. The problem with tracing is that it >> doesn't scale beyond some relatively small number of allocator events. > > I have looked at some of this functionality (heap profiling) but alas > it is not implemented yet. In addition the dtrace work appears to be > quite away from a workable solution with too many performance > penalties until some serious hacking is done. > > I am just not sure how to proceed, on one hand I do not really have > the skill to fix the /proc/pid/maps problem, nor figure out how to get > dtrace into the system in any time frame that is reasonable. > > All a few of us need is the addition of the trace back into the > existing utrace framework. > >>> Is it time to start installing with some form of debug symbols? This >>> would help us also with dtrace. >> Re: debug symbols, frame pointers, etc. necessary to make userland >> dtrace work by default, IMO we should strongly prefer such defaults. >> It's more reasonable to expect people who need every last bit of >> performance to remove functionality than to expect people who want to >> figure out what the system is doing to figure out what functionality >> to turn on. >> > > This is very true. I'm going to continue to work towards this end > with a few people and get up to speed on it so that hopefully we can > get to this point hopefully in the next release cycle or two. > > If you have a few moments, can you have a look at the "utrace2" > branches here: > https://github.com/alfredperlstein/freebsd/tree/utrace2 > > This branch contains the addition of the utrace2 system call which is > needed to structure data via utrace(2). The point of this is to avoid > kdump(1) needing to discern type of ktrace records based on arbitrary > size or other parameters and introduces an extensible protocol for new > types of utrace data. > > The utrace2 branch here augments jemalloc to use utrace2 to pass the > old utrace records, but in addition to pass the return address along > with the type and size of the allocation: > https://github.com/alfredperlstein/jemalloc/tree/utrace2 > > -Alfred Jason, Here are more convenient links that give diffs against FreeBSD and jemalloc for the proposed changes: FreeBSD: https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2 jemalloc: https://github.com/alfredperlstein/jemalloc/compare/master...utrace2 -Alfred