From owner-freebsd-questions@FreeBSD.ORG Mon Jul 28 04:14:05 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27D821065670 for ; Mon, 28 Jul 2008 04:14:05 +0000 (UTC) (envelope-from keramida@ceid.upatras.gr) Received: from igloo.linux.gr (igloo.linux.gr [62.1.205.36]) by mx1.freebsd.org (Postfix) with ESMTP id 9C6928FC24 for ; Mon, 28 Jul 2008 04:14:04 +0000 (UTC) (envelope-from keramida@ceid.upatras.gr) Received: from kobe.laptop (adsl121-117.kln.forthnet.gr [77.49.240.117]) (authenticated bits=128) by igloo.linux.gr (8.14.3/8.14.3/Debian-5) with ESMTP id m6S4Dq0r000732 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 28 Jul 2008 07:13:59 +0300 Received: from kobe.laptop (kobe.laptop [127.0.0.1]) by kobe.laptop (8.14.2/8.14.2) with ESMTP id m6S4DqYK009550; Mon, 28 Jul 2008 07:13:52 +0300 (EEST) (envelope-from keramida@ceid.upatras.gr) Received: (from keramida@localhost) by kobe.laptop (8.14.2/8.14.2/Submit) id m6S4Doll009549; Mon, 28 Jul 2008 07:13:50 +0300 (EEST) (envelope-from keramida@ceid.upatras.gr) From: Giorgos Keramidas To: vogelke+software@pobox.com References: <20080728015539.70030B7B9@kev.msw.wpafb.af.mil> Date: Mon, 28 Jul 2008 07:13:50 +0300 In-Reply-To: <20080728015539.70030B7B9@kev.msw.wpafb.af.mil> (Karl Vogel's message of "Sun, 27 Jul 2008 21:55:39 -0400 (EDT)") Message-ID: <87k5f6odc1.fsf@kobe.laptop> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-MailScanner-ID: m6S4Dq0r000732 X-Hellug-MailScanner: Found to be clean X-Hellug-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-3.798, required 5, autolearn=not spam, ALL_TRUSTED -1.80, AWL 0.60, BAYES_00 -2.60) X-Hellug-MailScanner-From: keramida@ceid.upatras.gr X-Spam-Status: No Cc: freebsd-questions@freebsd.org Subject: Re: malloc options X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2008 04:14:05 -0000 On Sun, 27 Jul 2008 21:55:39 -0400 (EDT), vogelke+software@pobox.com (Karl Vogel) wrote: >>> On Sat, 26 Jul 2008 17:36:35 -0700, >>> Doug Hardie wrote: > > D> The program has worked under considerable load for many years with > D> versions 3.7 to 6.2. Problems only occur with 7.0. The program is > D> quite complex and big. It uses probably hundreds of mallocs in a > D> typical use. The problems only occur reasonably randomly and only > D> under quite heavy load. The developer is looking into it, but the > D> problem only occurs on FreeBSD 7.0, not any other Unix systems. In > D> the meantime I am losing money because of it. > >>> On Sun, 27 Jul 2008 05:03:58 +0300, >>> Giorgos Keramidas said: > > G> While that's understandable, the current malloc() has undergone > G> quite extensive testing by Jason Evans and a lot of people who use > G> it in FreeBSD 7.X or later. Its ability to expose bugs in this way > G> was deemed important enough that it is now used by other projects > G> too. > > I ran into a similar problem with the BSD allocator running under > heavy load that didn't happen under any Solaris or Linux system I > used. I finally fixed it by using Doug Lea's malloc just for this one > application: > > http://shell.siscom.net/~vogelke/Software/Languages/C/Libraries/malloc/ > > This was under FreeBSD 6.*, but it might provide another data point if > you want to give it a try. I'm not sure how similar the two problems are. I quite frankly know _very_ little of what the original problem was, other than "I am encountering issues where values just seem to arbitrarily change". Memory exhaustion is a potential problem with almost any sort of allocator that fragments memory in any way, but random corruption of user data is probably a different issue :/ If you have some sort of description of the workload that triggered the memory exhaustion with jemalloc (the current malloc implementation in FreeBSD), it's probably a good idea to talk to Jason Evans about it (his email is "jasone" at FreeBSD.org). He may be able to help you tune malloc or even make changes to the system version of malloc that make it less vulnerable to this sort of problem.