From owner-freebsd-stable@FreeBSD.ORG Wed Mar 27 19:02:06 2013 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9CB2D144 for ; Wed, 27 Mar 2013 19:02:06 +0000 (UTC) (envelope-from ian@FreeBSD.org) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) by mx1.freebsd.org (Postfix) with ESMTP id 754C37C9 for ; Wed, 27 Mar 2013 19:02:06 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1UKvbp-000KwX-KU; Wed, 27 Mar 2013 19:02:05 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id r2RJ23cS011895; Wed, 27 Mar 2013 13:02:03 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1+7hAWijS6JeHCKOoCk77Ln Subject: Re: FreeBSD 9.1 excessive memory allocations [SOLVED] From: Ian Lepore To: Unga In-Reply-To: <1364409226.37379.YahooMailNeo@web161906.mail.bf1.yahoo.com> References: <1364322902.78474.YahooMailNeo@web161904.mail.bf1.yahoo.com> <1364393170.36972.49.camel@revolution.hippie.lan> <1364409226.37379.YahooMailNeo@web161906.mail.bf1.yahoo.com> Content-Type: text/plain; charset="us-ascii" Date: Wed, 27 Mar 2013 13:02:03 -0600 Message-ID: <1364410923.36972.67.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 19:02:06 -0000 On Wed, 2013-03-27 at 11:33 -0700, Unga wrote: > > ----- Original Message ----- > > > From: Ian Lepore > > To: Unga > > Cc: "freebsd-stable@freebsd.org" > > Sent: Wednesday, March 27, 2013 2:06 PM > > Subject: Re: FreeBSD 9.1 excessive memory allocations > > > > On Tue, 2013-03-26 at 11:35 -0700, Unga wrote: > >> Hi all > >> > >> > >> I have a heavily threaded C application, developed on an Intel Core i5 > > laptop (2 cores) running FreeBSD 8.1-RELEASE. > >> > >> When this application compile and run on another Intel Core i7 laptop (4 > > cores) running FreeBSD 9.1-RELEASE, this application immediately starts grabbing > > memory by over 100MB per second and soon exit with not enough RAM. > >> > >> > >> Both laptops having 4GB RAM. > >> > >> All malloc and free are mutex locked. > >> > >> Very rarely this problem happens on the i5 (2 cores) laptop too, but on the > > i7 laptop, it happens every time. > >> > >> Appreciate any feedback to identify and fix this issue. > >> > >> Best regards > >> Unga > >> > > > > Too many moving parts, you need to partition the problem. Is it the > > change in OS (and especially libraries) that causes the problem, or the > > change in the number of cores (more concurrent threads) is exposing some > > sort of application-side race condition? Given the fact that it does > > occur on 2 cores + freebsd 8.1, even if more rarely, it's almost surely > > an application problem. > > > > Perhaps you could use a tool such as valgrind to help track down the > > runaway allocations? > > > > Another way to expose thread race problems is to force more thread > > context switches. A blunt tool for doing so is to set hz=5000 or even > > higher in /boot/loader.conf. I've never done that on a desktop or > > laptop system, only on embedded systems, but it should still work okay. > > If changing the application code is easier, you can get a similar effect > > by creating a thread whose only job is to preempt other threads, by > > using rtprio_thread() to set it to real time priority, then just have it > > sleep for short random intervals (microseconds), all it does is wakes up > > and goes right back to sleep. > > > > Also, FYI, there's no need to use a mutex in your application to protect > > allocations. The memory allocator in libc is thread-safe, and in fact > > is particularly designed for efficient multi-threaded allocation. > > > > -- Ian > > > > Dear Tony, Alexander, Jan, Ian and Jeremy > > Thank you very much for your very valuable comments. > > Problem seems to be solved. But require more testing. > > 1. Fixed an application-level crucial bug. This is nearly a 7000 lines C app. It was really hard to see as the application is designed with 8 dedicated threads. > > 2. At run-time, this application shoots up to about 400 dynamic threads on the i7 machine, whereas on the i5 machine, it only shoots up 57 dynamic threads. It was bit scaring, therefore, made a hard limit on total number of threads to 64. > > Regarding mutex locks on allocations, as per the malloc(3), it says small and medium size allocations are done from per thread caches, therefore, thread-safe. My allocations are large in nature, about 5-7MB. > > Best regards > Unga I think you may be reading too much into the malloc manpage. When it mentions the use of per-thread small-object caches to avoid locking it's talking about performance, not thread safety. Allocations of all sizes are thread-safe, the library just assumes that huge allocations are rare enough that it doesn't use extra per-thread resources to avoid locking for them, it just uses locking for huge blocks. -- Ian