From owner-freebsd-stable@FreeBSD.ORG Wed Mar 27 18:53:56 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 13A9FBE4; Wed, 27 Mar 2013 18:53:56 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from cpsmtpb-ews04.kpnxchange.com (cpsmtpb-ews04.kpnxchange.com [213.75.39.7]) by mx1.freebsd.org (Postfix) with ESMTP id 7780071D; Wed, 27 Mar 2013 18:53:55 +0000 (UTC) Received: from cpsps-ews16.kpnxchange.com ([10.94.84.197]) by cpsmtpb-ews04.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 27 Mar 2013 19:52:47 +0100 Received: from CPSMTPM-TLF101.kpnxchange.com ([195.121.3.4]) by cpsps-ews16.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 27 Mar 2013 19:52:47 +0100 Received: from sjakie.klop.ws ([212.182.167.131]) by CPSMTPM-TLF101.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Wed, 27 Mar 2013 19:52:45 +0100 Received: from 212-182-167-131.ip.telfort.nl (localhost [127.0.0.1]) by sjakie.klop.ws (Postfix) with ESMTP id A42D541; Wed, 27 Mar 2013 19:52:45 +0100 (CET) Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: "Ian Lepore" , Unga Subject: Re: FreeBSD 9.1 excessive memory allocations [SOLVED] References: <1364322902.78474.YahooMailNeo@web161904.mail.bf1.yahoo.com> <1364393170.36972.49.camel@revolution.hippie.lan> <1364409226.37379.YahooMailNeo@web161906.mail.bf1.yahoo.com> Date: Wed, 27 Mar 2013 19:52:45 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <1364409226.37379.YahooMailNeo@web161906.mail.bf1.yahoo.com> User-Agent: Opera Mail/12.14 (FreeBSD) X-OriginalArrivalTime: 27 Mar 2013 18:52:45.0550 (UTC) FILETIME=[44E8A8E0:01CE2B1C] X-RcptDomain: freebsd.org Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 18:53:56 -0000 On Wed, 27 Mar 2013 19:33:46 +0100, Unga wrote: > > > ----- Original Message ----- > >> From: Ian Lepore >> To: Unga >> Cc: "freebsd-stable@freebsd.org" >> Sent: Wednesday, March 27, 2013 2:06 PM >> Subject: Re: FreeBSD 9.1 excessive memory allocations >> >> On Tue, 2013-03-26 at 11:35 -0700, Unga wrote: >>> Hi all >>> >>> >>> I have a heavily threaded C application, developed on an Intel Core i5 >> laptop (2 cores) running FreeBSD 8.1-RELEASE. >>> >>> When this application compile and run on another Intel Core i7 laptop >>> (4 >> cores) running FreeBSD 9.1-RELEASE, this application immediately starts >> grabbing >> memory by over 100MB per second and soon exit with not enough RAM. >>> >>> >>> Both laptops having 4GB RAM. >>> >>> All malloc and free are mutex locked. >>> >>> Very rarely this problem happens on the i5 (2 cores) laptop too, but >>> on the >> i7 laptop, it happens every time. >>> >>> Appreciate any feedback to identify and fix this issue. >>> >>> Best regards >>> Unga >>> >> >> Too many moving parts, you need to partition the problem. Is it the >> change in OS (and especially libraries) that causes the problem, or the >> change in the number of cores (more concurrent threads) is exposing some >> sort of application-side race condition? Given the fact that it does >> occur on 2 cores + freebsd 8.1, even if more rarely, it's almost surely >> an application problem. >> Perhaps you could use a tool such as valgrind to help track down the >> runaway allocations? >> >> Another way to expose thread race problems is to force more thread >> context switches. A blunt tool for doing so is to set hz=5000 or even >> higher in /boot/loader.conf. I've never done that on a desktop or >> laptop system, only on embedded systems, but it should still work okay. >> If changing the application code is easier, you can get a similar effect >> by creating a thread whose only job is to preempt other threads, by >> using rtprio_thread() to set it to real time priority, then just have it >> sleep for short random intervals (microseconds), all it does is wakes up >> and goes right back to sleep. >> >> Also, FYI, there's no need to use a mutex in your application to protect >> allocations. The memory allocator in libc is thread-safe, and in fact >> is particularly designed for efficient multi-threaded allocation. >> >> -- Ian >> > > Dear Tony, Alexander, Jan, Ian and Jeremy > > Thank you very much for your very valuable comments. > > Problem seems to be solved. But require more testing. > > 1. Fixed an application-level crucial bug. This is nearly a 7000 lines C > app. It was really hard to see as the application is designed with 8 > dedicated threads. > > 2. At run-time, this application shoots up to about 400 dynamic threads > on the i7 machine, whereas on the i5 machine, it only shoots up 57 > dynamic threads. It was bit scaring, therefore, made a hard limit on > total number of threads to 64. > > Regarding mutex locks on allocations, as per the malloc(3), it says > small and medium size allocations are done from per thread caches, > therefore, thread-safe. My allocations are large in nature, about 5-7MB. The per thread caches are for speed. Not for thread-safety. Ronald.