Date: Sun, 6 Mar 2011 20:56:11 -0600 From: Eric Anderson <anderson@ttel.com> To: Vlad Galu <dudu@dudu.ro> Cc: Deomid Ryabkov <myself@rojer.pp.ru>, freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? Message-ID: <79592F5D-A099-4474-A818-4DD4C6F279D5@ttel.com> In-Reply-To: <AANLkTi=K9QuK%2BbyxF1t7HzG9wKxXek804tNOgzM3Dp_b@mail.gmail.com> References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> <AANLkTikd22VT3CVtvkGdw6zos%2Biq4Ve8y1W_WLr0qctS@mail.gmail.com> <AANLkTi=K9QuK%2BbyxF1t7HzG9wKxXek804tNOgzM3Dp_b@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mar 6, 2011, at 6:09 AM, Vlad Galu wrote: >=20 >=20 > On Sun, Mar 6, 2011 at 2:06 PM, Vlad Galu <dudu@dudu.ro> wrote: >=20 >=20 > On Sun, Mar 6, 2011 at 6:53 AM, Eric Anderson <anderson@ttel.com> = wrote: > On Mar 5, 2011, at 10:44 AM, Deomid Ryabkov wrote: >=20 > > On 03/05/2011 04:02 AM, Eric Anderson wrote: > >> Hi all, > >> > >> I have a moderately threaded userland program (all C) I am working = on (using pthreads, freebsd 8.1 64bit). It seems to leak memory (using = standard malloc/free) badly. > > as opposed to what? OpenBSD? Linux? Windows? why do you think your = problem is specific to FreeBSD (as evidenced by your post to a = FreeBSD-specific list) or is related to threaded programs? >=20 > OpenSolaris and Mac OS X. I didn't really assume or state it was = specific to FreeBSD, just that this scenario was on FreeBSD. I happen to = do most of the development and testing on OS X and FreeBSD, and I've = enjoyed the FreeBSD community for a very long time. >=20 >=20 >=20 > > > >> I am using pcap to capture packets and process them. I have a = handful of libs statically linked in (pcap is one, the rest don't seem = to matter - I can remove them and still see the leak). > >> > >> Does anyone know of issues regarding malloc/free on multithreaded = userland apps? > > hell yeah. it goes like this: you malloc() then forget to free() - = boom, you have a memory leak. > > > > you're welcome. >=20 >=20 > Thanks, very insightful. >=20 >=20 > > > > > > sarcasm aside, those questions still remain: why do you think = os/libraries are the problem and not your code? >=20 > Because I am tracking all malloc and free calls within the application = code (aside from libraries) and I can account for all malloc'ed memory = and freed memory in both count and by bytes, yet looking at ps output = shows a very different story, and if I leave it run long enough, will = consume all memory and swap in the system and then be killed off. I = wrapped malloc/free in a function, and record all memory alloc'ed and = free'd. The only memory I cannot track is memory alloced and freed by = libraries I am pulling in (well, can't track easily anyway without = hacking through all of their source code). >=20 >=20 >=20 > > you can't post all of it, ok, and we don't want all of it either. = can you isolate a specific example of where valid usage of a library = causes a leak? >=20 >=20 > Not really - if I could, I would have fixed it by now. It's a = non-trivial issue - which is why I am beginning to suspect something = more complicated than a "oops I forgot to free" kind of error. Plus, I = have seen a few people elsewhere on various forums/mailing lists with = similar issues claiming that switching to the Hoard allocator fixed the = problem (which doesn't seem to be happy with 32bit FreeBSD - tried it). = I have also seen various comments about pthreads and memory allocators = having apparent leaks at some threading level, but not sure. >=20 > Thanks, > Eric >=20 >=20 > Had there been a memleak in jemalloc, I'm sure more people would have = spotted it by now. How many pcap_t structs do you use in your app? = libpcap is not threadsafe. FWIW I've been running a pcap/threaded = application continuously for the past couple of years and the memory = usage has been constant. >=20 > Also, put a couple of printf()s before and after = pcap_dispatch()/pcap_loop() (if you use any of them), to make sure they = don't block waiting for your callback to return (on some platforms it = doesn't, so you never get to free memory in outer frames). This might = not necessarily be your case, but it's worth taking a look at. > =20 >=20 > That should've been pcap_dispatch()/pcap_next(). Hmm - interestingly enough, I use a third party lib that handles some = parts of what pcap does.. I'll did into that lib and check these = things. Thanks for the idea! Eric
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?79592F5D-A099-4474-A818-4DD4C6F279D5>