From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 04:53:58 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E8E1106566B for ; Sun, 6 Mar 2011 04:53:58 +0000 (UTC) (envelope-from anderson@ttel.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id D57538FC08 for ; Sun, 6 Mar 2011 04:53:57 +0000 (UTC) Received: by yie12 with SMTP id 12so1419612yie.13 for ; Sat, 05 Mar 2011 20:53:57 -0800 (PST) Received: by 10.236.184.130 with SMTP id s2mr826783yhm.71.1299387236937; Sat, 05 Mar 2011 20:53:56 -0800 (PST) Received: from [10.0.7.145] (r74-193-70-198.pfvlcmta01.grtntx.tl.dh.suddenlink.net [74.193.70.198]) by mx.google.com with ESMTPS id u49sm820073yhn.37.2011.03.05.20.53.55 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 05 Mar 2011 20:53:56 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Eric Anderson In-Reply-To: <4D726887.5080800@rojer.pp.ru> Date: Sat, 5 Mar 2011 22:53:54 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> To: Deomid Ryabkov X-Mailer: Apple Mail (2.1082) X-Mailman-Approved-At: Sun, 06 Mar 2011 06:40:58 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 04:53:58 -0000 On Mar 5, 2011, at 10:44 AM, Deomid Ryabkov wrote: > On 03/05/2011 04:02 AM, Eric Anderson wrote: >> Hi all, >>=20 >> I have a moderately threaded userland program (all C) I am working on = (using pthreads, freebsd 8.1 64bit). It seems to leak memory (using = standard malloc/free) badly. > as opposed to what? OpenBSD? Linux? Windows? why do you think your = problem is specific to FreeBSD (as evidenced by your post to a = FreeBSD-specific list) or is related to threaded programs? OpenSolaris and Mac OS X. I didn't really assume or state it was = specific to FreeBSD, just that this scenario was on FreeBSD. I happen to = do most of the development and testing on OS X and FreeBSD, and I've = enjoyed the FreeBSD community for a very long time. >=20 >> I am using pcap to capture packets and process them. I have a = handful of libs statically linked in (pcap is one, the rest don't seem = to matter - I can remove them and still see the leak). >>=20 >> Does anyone know of issues regarding malloc/free on multithreaded = userland apps? > hell yeah. it goes like this: you malloc() then forget to free() - = boom, you have a memory leak. >=20 > you're welcome. Thanks, very insightful. >=20 >=20 > sarcasm aside, those questions still remain: why do you think = os/libraries are the problem and not your code? Because I am tracking all malloc and free calls within the application = code (aside from libraries) and I can account for all malloc'ed memory = and freed memory in both count and by bytes, yet looking at ps output = shows a very different story, and if I leave it run long enough, will = consume all memory and swap in the system and then be killed off. I = wrapped malloc/free in a function, and record all memory alloc'ed and = free'd. The only memory I cannot track is memory alloced and freed by = libraries I am pulling in (well, can't track easily anyway without = hacking through all of their source code). > you can't post all of it, ok, and we don't want all of it either. can = you isolate a specific example of where valid usage of a library causes = a leak? Not really - if I could, I would have fixed it by now. It's a = non-trivial issue - which is why I am beginning to suspect something = more complicated than a "oops I forgot to free" kind of error. Plus, I = have seen a few people elsewhere on various forums/mailing lists with = similar issues claiming that switching to the Hoard allocator fixed the = problem (which doesn't seem to be happy with 32bit FreeBSD - tried it). = I have also seen various comments about pthreads and memory allocators = having apparent leaks at some threading level, but not sure. Thanks, Eric