From owner-freebsd-current@FreeBSD.ORG Wed Mar 22 15:20:06 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 56D3E16A427; Wed, 22 Mar 2006 15:20:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id 701A643D55; Wed, 22 Mar 2006 15:20:05 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from localhost (john@localhost [127.0.0.1]) by server.baldwin.cx (8.13.4/8.13.4) with ESMTP id k2MFJxGn019515; Wed, 22 Mar 2006 10:20:02 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-current@freebsd.org Date: Wed, 22 Mar 2006 10:19:39 -0500 User-Agent: KMail/1.9.1 References: <20060321184019.GX35129@funkthat.com> <1EB2EEE3-855C-4B76-81A6-1880526797CE@freebsd.org> <44215B1B.1080104@mac.com> In-Reply-To: <44215B1B.1080104@mac.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200603221019.43713.jhb@freebsd.org> X-Virus-Scanned: ClamAV 0.87.1/1350/Wed Mar 22 05:12:47 2006 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-3.7 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on server.baldwin.cx Cc: John-Mark Gurney , Jason Evans Subject: Re: core dumps are HUGE... X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 15:20:06 -0000 On Wednesday 22 March 2006 09:11, Chuck Swiger wrote: > Hi, Jason-- > > First, thanks for your work on the new jemalloc. > > Jason Evans wrote: > > On Mar 21, 2006, at 2:20 PM, John-Mark Gurney wrote: > [ ... ] > >> And yes, I'm pretty sure that I have a world from before your reduction > >> in chunk size... Having a minimum of a 6meg core doesn't sound like a > >> good idea... It's definately not going to play nice with small systems.. > > > > I don't think that a 6 MB core file is the big deal you are making it > > out to be. A 50 MB core file is a pain, mainly because it takes a long > > time to dump core. > > The smallest processes that people run often are likely going to be /bin/sh and > the typical fodder of shell scripts (grep, awk, and so forth). Under 4.x, > /bin/sh tended to have VSIZE of 640K and RSS of ~300K; under 5.x and later, > /bin/sh has VSIZE of 1.6MB and RSS of ~1MB. > > Creating a 50MB corefile from a 2MB VSIZE process is unreasonable. Creating a > 6MB corefile from a 2MB VSIZE process would be reasonable but probably could be > improved by a factor of two. > > On the other hand, the staticly linked "t" program is: > > chuck 62964 0.0 0.1 232 128 p0 TX 8:09AM 0:00.03 /tmp/t > > 128K RSS, 232 VSIZE. A 6MB corefile is a factor of 25 larger, which strikes me > as something that could be improved by more than a factor of ten. > > What else is being dumped that's not counted by VSIZE, aside from a header and > maybe some per-thread state, (although aren't the per-thread stacks already part > of VSIZE)...? > > Can jemalloc only create per-CPU arenas only for processes which are themselves > multithreaded, when it's running on a multi-CPU system? Would that help reduce > the amount of allocated but unreferenced memory that is involved for the common > case of /bin/sh and friends? I think the better path is to provide sparse coredumps. I.e., when dumping a core, leave the parts of the process map that are mapped but have no backing store yet (b/c the pages haven't been touched) sparse by not writing to them, but just seeking past them. This doesn't require complicating the malloc implementation just for the sake of a core dump on a CF device. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org