From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 12:09:57 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95CD9106566B for ; Sun, 6 Mar 2011 12:09:57 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5841D8FC12 for ; Sun, 6 Mar 2011 12:09:56 +0000 (UTC) Received: by qwj8 with SMTP id 8so2862305qwj.13 for ; Sun, 06 Mar 2011 04:09:56 -0800 (PST) Received: by 10.224.8.205 with SMTP id i13mr338732qai.214.1299413396268; Sun, 06 Mar 2011 04:09:56 -0800 (PST) MIME-Version: 1.0 Received: by 10.224.46.161 with HTTP; Sun, 6 Mar 2011 04:09:16 -0800 (PST) In-Reply-To: References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> From: Vlad Galu Date: Sun, 6 Mar 2011 14:09:16 +0200 Message-ID: To: Eric Anderson Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Deomid Ryabkov , freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 12:09:57 -0000 On Sun, Mar 6, 2011 at 2:06 PM, Vlad Galu wrote: > > > On Sun, Mar 6, 2011 at 6:53 AM, Eric Anderson wrote: > >> On Mar 5, 2011, at 10:44 AM, Deomid Ryabkov wrote: >> >> > On 03/05/2011 04:02 AM, Eric Anderson wrote: >> >> Hi all, >> >> >> >> I have a moderately threaded userland program (all C) I am working on >> (using pthreads, freebsd 8.1 64bit). It seems to leak memory (using >> standard malloc/free) badly. >> > as opposed to what? OpenBSD? Linux? Windows? why do you think your >> problem is specific to FreeBSD (as evidenced by your post to a >> FreeBSD-specific list) or is related to threaded programs? >> >> OpenSolaris and Mac OS X. I didn't really assume or state it was specific >> to FreeBSD, just that this scenario was on FreeBSD. I happen to do most of >> the development and testing on OS X and FreeBSD, and I've enjoyed the >> FreeBSD community for a very long time. >> >> >> >> > >> >> I am using pcap to capture packets and process them. I have a handful >> of libs statically linked in (pcap is one, the rest don't seem to matter - I >> can remove them and still see the leak). >> >> >> >> Does anyone know of issues regarding malloc/free on multithreaded >> userland apps? >> > hell yeah. it goes like this: you malloc() then forget to free() - boom, >> you have a memory leak. >> > >> > you're welcome. >> >> >> Thanks, very insightful. >> >> >> > >> > >> > sarcasm aside, those questions still remain: why do you think >> os/libraries are the problem and not your code? >> >> Because I am tracking all malloc and free calls within the application >> code (aside from libraries) and I can account for all malloc'ed memory and >> freed memory in both count and by bytes, yet looking at ps output shows a >> very different story, and if I leave it run long enough, will consume all >> memory and swap in the system and then be killed off. I wrapped malloc/free >> in a function, and record all memory alloc'ed and free'd. The only memory I >> cannot track is memory alloced and freed by libraries I am pulling in (well, >> can't track easily anyway without hacking through all of their source code). >> >> >> >> > you can't post all of it, ok, and we don't want all of it either. can >> you isolate a specific example of where valid usage of a library causes a >> leak? >> >> >> Not really - if I could, I would have fixed it by now. It's a non-trivial >> issue - which is why I am beginning to suspect something more complicated >> than a "oops I forgot to free" kind of error. Plus, I have seen a few >> people elsewhere on various forums/mailing lists with similar issues >> claiming that switching to the Hoard allocator fixed the problem (which >> doesn't seem to be happy with 32bit FreeBSD - tried it). I have also seen >> various comments about pthreads and memory allocators having apparent leaks >> at some threading level, but not sure. >> >> Thanks, >> Eric >> >> > Had there been a memleak in jemalloc, I'm sure more people would have > spotted it by now. How many pcap_t structs do you use in your app? libpcap > is not threadsafe. FWIW I've been running a pcap/threaded application > continuously for the past couple of years and the memory usage has been > constant. > > Also, put a couple of printf()s before and after > pcap_dispatch()/pcap_loop() (if you use any of them), to make sure they > don't block waiting for your callback to return (on some platforms it > doesn't, so you never get to free memory in outer frames). This might not > necessarily be your case, but it's worth taking a look at. > > That should've been pcap_dispatch()/pcap_next(). > >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org >> " >> > > > > -- > Good, fast & cheap. Pick any two. > -- Good, fast & cheap. Pick any two.