From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 23 17:28:53 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CD0B1C97; Sun, 23 Dec 2012 17:28:53 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from canonware.com (canonware.com [204.109.63.53]) by mx1.freebsd.org (Postfix) with ESMTP id A78D58FC0A; Sun, 23 Dec 2012 17:28:53 +0000 (UTC) Received: from [192.168.168.12] (70-91-206-178-BusName-SFBA.hfc.comcastbusiness.net [70.91.206.178]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by canonware.com (Postfix) with ESMTPSA id DE4E628417; Sun, 23 Dec 2012 09:28:52 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: malloc+utrace, tracking memory leaks in a running program. From: Jason Evans In-Reply-To: <50D52B10.1060205@mu.org> Date: Sun, 23 Dec 2012 09:28:52 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <50D52B10.1060205@mu.org> To: Alfred Perlstein X-Mailer: Apple Mail (2.1499) Cc: Daniel Eischen , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2012 17:28:53 -0000 On Dec 21, 2012, at 7:37 PM, Alfred Perlstein wrote: > So the other day in an effort to debug a memory leak I decided to take = a look at malloc+utrace(2) and decided to make a tool to debug where = leaks are coming from. >=20 > A few hours later I have: > 1) a new version of utrace(2) (utrace2(2)) that uses structured data = to prevent overloading of data. (utrace2.diff) > 2) changes to ktrace and kdump to decode the new format. (also in = utrace2.diff) > 3) changes to jemalloc to include the new format AND the function = caller so it's easy to get the source of the leaks. (also in = utrace2.diff) > 4) a program that can take a pipe of kdump(1) and figure out what = memory has leaked. (alloctrace.py) > 5) simple test program (test_utrace.c) >=20 > [=85] Have you looked at the heap profiling functionality built into jemalloc? = It's not currently enabled on FreeBSD, but as far as I know, the only = issue keeping it from being useful is the absence of a Linux-compatible = /proc//maps (and the gperftools folks may already have a solution = for that; I haven't looked). I think it makes more sense to get that = sorted out than to develop a separate trace-based leak checker. The = problem with tracing is that it doesn't scale beyond some relatively = small number of allocator events. > Is it time to start installing with some form of debug symbols? This = would help us also with dtrace. Re: debug symbols, frame pointers, etc. necessary to make userland = dtrace work by default, IMO we should strongly prefer such defaults. = It's more reasonable to expect people who need every last bit of = performance to remove functionality than to expect people who want to = figure out what the system is doing to figure out what functionality to = turn on. Thanks, Jason=