From owner-freebsd-stable@FreeBSD.ORG Wed Mar 27 14:15:16 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2BF654F6 for ; Wed, 27 Mar 2013 14:15:16 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta12.emeryville.ca.mail.comcast.net (qmta12.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:44:76:96:27:227]) by mx1.freebsd.org (Postfix) with ESMTP id 0F7B8170 for ; Wed, 27 Mar 2013 14:15:16 +0000 (UTC) Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89]) by qmta12.emeryville.ca.mail.comcast.net with comcast id Gdga1l0041vN32cACeFF8N; Wed, 27 Mar 2013 14:15:15 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta22.emeryville.ca.mail.comcast.net with comcast id GeFF1l0011t3BNj8ieFFfq; Wed, 27 Mar 2013 14:15:15 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id DD0E173A1C; Wed, 27 Mar 2013 07:15:14 -0700 (PDT) Date: Wed, 27 Mar 2013 07:15:14 -0700 From: Jeremy Chadwick To: Ian Lepore Subject: Re: FreeBSD 9.1 excessive memory allocations Message-ID: <20130327141514.GA9855@icarus.home.lan> References: <1364322902.78474.YahooMailNeo@web161904.mail.bf1.yahoo.com> <1364393170.36972.49.camel@revolution.hippie.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1364393170.36972.49.camel@revolution.hippie.lan> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1364393715; bh=cpOE2woHnXDN1ebgI0J7SrA59dF0F0Ogkr8rPEy6yrU=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=ciLILPknRafy4hO3nZqjhA1XShRh0vtjon3frEqSHft6Vc/tCWuE2exahiNBRF3Ur AIlaIS9TRZq9oDsoowAnSUNdNVsZSsyZzlMruljD0w54dIhecv/ry/T1WejpezUvg0 wJBp+gUsO5c46HzFnIW8g5bJ9an1xxfThm/u2kX0PjHWLN7XpPD0Yu4xKpXxigrbax 7NgOlPl7CgqkbsBdqQhn9/5cXiMSrk2C2499Hta9jvnYP8FH9KNx/bbDjgOLRCLfZD swf45SXYXNtQplvLxsaYeftQMF5/6R69BN2GIPE9uVaBCdUW+Kz6AuetZdaXD1Wq8P SSumMhVVuRsJA== Cc: Unga , "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 14:15:16 -0000 On Wed, Mar 27, 2013 at 08:06:10AM -0600, Ian Lepore wrote: > On Tue, 2013-03-26 at 11:35 -0700, Unga wrote: > > Hi all > > > > > > I have a heavily threaded C application, developed on an Intel Core i5 laptop (2 cores) running FreeBSD 8.1-RELEASE. > > > > When this application compile and run on another Intel Core i7 laptop (4 cores) running FreeBSD 9.1-RELEASE, this application immediately starts grabbing memory by over 100MB per second and soon exit with not enough RAM. > > > > > > Both laptops having 4GB RAM. > > > > All malloc and free are mutex locked. > > > > Very rarely this problem happens on the i5 (2 cores) laptop too, but on the i7 laptop, it happens every time. > > > > Appreciate any feedback to identify and fix this issue. > > > > Best regards > > Unga > > > > Too many moving parts, you need to partition the problem. Is it the > change in OS (and especially libraries) that causes the problem, or the > change in the number of cores (more concurrent threads) is exposing some > sort of application-side race condition? Given the fact that it does > occur on 2 cores + freebsd 8.1, even if more rarely, it's almost surely > an application problem. > > Perhaps you could use a tool such as valgrind to help track down the > runaway allocations? > > Another way to expose thread race problems is to force more thread > context switches. A blunt tool for doing so is to set hz=5000 or even > higher in /boot/loader.conf. I've never done that on a desktop or > laptop system, only on embedded systems, but it should still work okay. > If changing the application code is easier, you can get a similar effect > by creating a thread whose only job is to preempt other threads, by > using rtprio_thread() to set it to real time priority, then just have it > sleep for short random intervals (microseconds), all it does is wakes up > and goes right back to sleep. > > Also, FYI, there's no need to use a mutex in your application to protect > allocations. The memory allocator in libc is thread-safe, and in fact > is particularly designed for efficient multi-threaded allocation. Thank you Ian -- you've covered nearly all of the points I wanted to mention in a reply, but opted to not send it (one of those "after I read it I had second thoughts" situations). For the OP: I *strongly* recommend ports/devel/valgrind as mentioned. It can be a little bit daunting getting started with it ("okay, wow, what am I doing with this thing?"), but it's very useful for situations exactly like this. There's lots of random tidbits of info on it on the web too. If the issue turns out to be something in userland (library-wise) or the kernel, there are FreeBSD.org people who can help with those (I can think of 4 off the top of my head right now). They can be CC'd if things get to that point, but exhaust other avenues first. It would also be wise, at that time, if you could make the source to the application available, that would probably help folks narrow down what's happening -- otherwise they'll be forced to rely entirely on claims like "it *should* be behaving like this", which isn't necessarily the same thing as how the underlying code bits actually behave. My impression as of this writing, is that the problem is in the application. If FreeBSD 9.1 had major problems with memory leaks in userland threaded apps, I'd expect it would have been discovered by now; that said, there is always the chance there is a bug somewhere outside of your application code, it just seems slim. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |