From owner-freebsd-stable@FreeBSD.ORG Wed Mar 27 18:33:53 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 101DC5FB for ; Wed, 27 Mar 2013 18:33:53 +0000 (UTC) (envelope-from unga888@yahoo.com) Received: from nm1.bullet.mail.bf1.yahoo.com (nm1.bullet.mail.bf1.yahoo.com [98.139.212.160]) by mx1.freebsd.org (Postfix) with SMTP id B74095EA for ; Wed, 27 Mar 2013 18:33:52 +0000 (UTC) Received: from [98.139.215.142] by nm1.bullet.mail.bf1.yahoo.com with NNFMP; 27 Mar 2013 18:33:46 -0000 Received: from [98.139.212.251] by tm13.bullet.mail.bf1.yahoo.com with NNFMP; 27 Mar 2013 18:33:46 -0000 Received: from [127.0.0.1] by omp1060.mail.bf1.yahoo.com with NNFMP; 27 Mar 2013 18:33:46 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 792227.67692.bm@omp1060.mail.bf1.yahoo.com Received: (qmail 52094 invoked by uid 60001); 27 Mar 2013 18:33:46 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1364409226; bh=WFtnnDSfjN3M4t9fQfAMEFjNeVjDkELGMnZ52zao7EE=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=kClBp80Hdl029DgefobTxu0S+0+Sk0QUVSJ8GvpofZYGG0E4q+FiEDcffRiTJanN9j1hIwWEprVHDKVL/9oj+Kb1eEbbvidfaG7LXy+vLDz/eZNiql0tMN2INFe44b/WiN4CrW1TAsATJ0MG77TT+K6ylNeuavv6r/bQmEdFAEo= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=vApUcNamtqfzwWzomxpCgMYXd4aouVVR5lxigLZGQVKu/b5mE4I7J1mLcvI2JUcAj7ok+5SRPeVKGXArFU6xh3YNNKRMCrRdJHlqB+OYGQusZL1Rm/ctGOfOJxQfxk9b2CPVraG201MwXcvaYkWs/5W8vTSJFrXFwi+n7Mi3LPg=; X-YMail-OSG: rkwxvyEVM1nuJi1Q.yhWDKULqzxFa4rXo6czW0efGe7cU.f kC9.e__oIwDlov2zFIo1_0k573CqLEJ_I9E3FZGEIDfOlPLSPJFOJHhzHNCw 4WPnFPKY3k2WCUneYwWzIRF1D5dCTeY_LE06uToMz4T8301_ib__e08JOA25 rQ_v6KbOtTQTfZUE7EQNhKWX9ZH9eMEmbGFIcdSHnPSkwu.UmyQ4SpQ4qzXk Hf_b1OXZnKXPTfRCtqpk.A8Joc42LninBqr5.p2PDSjLvn2shyWhr.dXxlw0 V5JigUPuxpsnhPdsaoCJSMXqiYJRX.POZZ50k9mqGe8ziKrH2lwhaiW_TmFU P79MzhH.3bQSPVsyj164yNKYtBagATe0a7d6pSwI2z6EoTbNGbf1g1BJeRNs bOk024Sy.7CFweZeY8AjacJ2SHd_INoxepaDVOrZDHY.WKRczpYrx1T69C_8 wufpvoBtFQW.29w2tGwM8PDL7KM9BJ8jiwIMeCZX_ClVIaYvNyQ7pG_MKxFg nOeEDZ1Di6RA.mg8CXA-- Received: from [112.134.135.82] by web161906.mail.bf1.yahoo.com via HTTP; Wed, 27 Mar 2013 11:33:46 PDT X-Rocket-MIMEInfo: 002.001, CgotLS0tLSBPcmlnaW5hbCBNZXNzYWdlIC0tLS0tCgo.IEZyb206IElhbiBMZXBvcmUgPGlhbkBGcmVlQlNELm9yZz4KPiBUbzogVW5nYSA8dW5nYTg4OEB5YWhvby5jb20.Cj4gQ2M6ICJmcmVlYnNkLXN0YWJsZUBmcmVlYnNkLm9yZyIgPGZyZWVic2Qtc3RhYmxlQEZyZWVCU0Qub3JnPgo.IFNlbnQ6IFdlZG5lc2RheSwgTWFyY2ggMjcsIDIwMTMgMjowNiBQTQo.IFN1YmplY3Q6IFJlOiBGcmVlQlNEIDkuMSBleGNlc3NpdmUgbWVtb3J5IGFsbG9jYXRpb25zCj4gCj4gT24gVHVlLCAyMDEzLTAzLTI2IGF0IDEBMAEBAQE- X-Mailer: YahooMailWebService/0.8.139.530 References: <1364322902.78474.YahooMailNeo@web161904.mail.bf1.yahoo.com> <1364393170.36972.49.camel@revolution.hippie.lan> Message-ID: <1364409226.37379.YahooMailNeo@web161906.mail.bf1.yahoo.com> Date: Wed, 27 Mar 2013 11:33:46 -0700 (PDT) From: Unga Subject: Re: FreeBSD 9.1 excessive memory allocations [SOLVED] To: Ian Lepore In-Reply-To: <1364393170.36972.49.camel@revolution.hippie.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Unga List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 18:33:53 -0000 =0A=0A----- Original Message -----=0A=0A> From: Ian Lepore =0A> To: Unga =0A> Cc: "freebsd-stable@freebsd.org" =0A> Sent: Wednesday, March 27, 2013 2:06 PM=0A> S= ubject: Re: FreeBSD 9.1 excessive memory allocations=0A> =0A> On Tue, 2013-= 03-26 at 11:35 -0700, Unga wrote:=0A>> Hi all=0A>> =0A>> =0A>> I have a h= eavily threaded C application, developed on an Intel Core i5 =0A> laptop (2= cores) running FreeBSD 8.1-RELEASE.=0A>> =0A>> When this application comp= ile and run on another Intel Core i7 laptop (4 =0A> cores) running FreeBSD = 9.1-RELEASE, this application immediately starts grabbing =0A> memory by ov= er 100MB per second and soon exit with not enough RAM.=0A>> =0A>> =0A>> Bo= th laptops having 4GB RAM.=0A>> =0A>> All malloc and free are mutex locked= .=0A>> =0A>> Very rarely this problem happens on the i5 (2 cores) laptop t= oo, but on the =0A> i7 laptop, it happens every time.=0A>> =0A>> Appreciat= e any feedback to identify and fix this issue.=0A>> =0A>> Best regards=0A>= > Unga=0A>> =0A> =0A> Too many moving parts, you need to partition the pro= blem.=A0 Is it the=0A> change in OS (and especially libraries) that causes = the problem, or the=0A> change in the number of cores (more concurrent thre= ads) is exposing some=0A> sort of application-side race condition?=A0 Given= the fact that it does=0A> occur on 2 cores + freebsd 8.1, even if more rar= ely, it's almost surely=0A> an application problem.=A0 =0A> =0A> Perhaps yo= u could use a tool such as valgrind to help track down the=0A> runaway allo= cations?=0A> =0A> Another way to expose thread race problems is to force mo= re thread=0A> context switches.=A0 A blunt tool for doing so is to set hz= =3D5000 or even=0A> higher in /boot/loader.conf.=A0 I've never done that on= a desktop or=0A> laptop system, only on embedded systems, but it should st= ill work okay.=0A> If changing the application code is easier, you can get = a similar effect=0A> by creating a thread whose only job is to preempt othe= r threads, by=0A> using rtprio_thread() to set it to real time priority, th= en just have it=0A> sleep for short random intervals (microseconds), all it= does is wakes up=0A> and goes right back to sleep.=0A> =0A> Also, FYI, the= re's no need to use a mutex in your application to protect=0A> allocations.= =A0 The memory allocator in libc is thread-safe, and in fact=0A> is particu= larly designed for efficient multi-threaded allocation.=0A> =0A> -- Ian=0A>= =0A=0ADear Tony, Alexander, Jan, Ian and Jeremy=0A=0AThank you very much fo= r your very valuable comments.=0A=0AProblem seems to be solved. But require= more testing.=0A=0A1. Fixed an application-level crucial bug. This is near= ly a 7000 lines C app. It was really hard to see as the application is desi= gned with 8 dedicated threads.=0A=0A2. At run-time, this application shoots= up to about 400 dynamic threads on the i7 machine, whereas on the i5 machi= ne, it only shoots up 57 dynamic threads. It was bit scaring, therefore, ma= de a hard limit on total number of threads to 64.=0A=0ARegarding mutex lock= s on allocations, as per the malloc(3), it says small and medium size alloc= ations are done from per thread caches, therefore, thread-safe. My allocati= ons are large in nature, about 5-7MB.=0A=0ABest regards=0AUnga