From owner-freebsd-current@FreeBSD.ORG Fri Jan 19 00:34:56 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D548916A49E for ; Fri, 19 Jan 2007 00:34:56 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from canonware.com (24-38-119-150-st.losaca.adelphia.net [24.38.119.150]) by mx1.freebsd.org (Postfix) with ESMTP id B100D13C428 for ; Fri, 19 Jan 2007 00:34:56 +0000 (UTC) (envelope-from jasone@FreeBSD.org) Received: from [127.0.0.1] (24-38-119-150-st.losaca.adelphia.net [24.38.119.150]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by canonware.com (Postfix) with ESMTP id 528831298C0; Thu, 18 Jan 2007 16:09:12 -0800 (PST) Message-ID: <45B00BF5.6030200@FreeBSD.org> Date: Thu, 18 Jan 2007 16:08:21 -0800 From: Jason Evans User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: Stefan Ehmann References: <200701172045.35137.shoesoft@gmx.net> In-Reply-To: <200701172045.35137.shoesoft@gmx.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-current@freebsd.org Subject: Re: very high memory usage in -current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Jan 2007 00:34:56 -0000 Stefan Ehmann wrote: > [huge jpeg/png images cause memory bloat with konqueror and gtk apps, on -CURRENT.] > > I don't really know what's causing this problem. Maybe it's related to > jemalloc, but I'd be surprised if no one else has noticed this before. > > size/res values reported by top > > 6.2 desktop pc with nvidia binary driver > konqueror 40M/30M > > CURRENT notebook with the i810 driver > konqueror 290M/250M > > Any clues or more information needed? If you think jemalloc is involved, the easiest way to check is by reverting src/lib/libc/stdlib/malloc.c to revision 1.92, which is phkmalloc. If this substantially changes memory usage, then there are further diagnostics that can be used to help understand the issue. ----- If jemalloc is involved, here's what could cause such behavior. First, you would have to be running on a 32-bit platform, so that sbrk() is in use (rather than pure mmap() as for the 64-bit platforms). 1) The application would allocate some large chunks of memory for image processing. These data would be allocated in the data segment. 2) The application would allocate some additional long-lived object for which no existing memory region could be recycled. This would cause the object to be allocated in the data segment, at the end of the sbrk()ed memory. 3) The application would free most/all of the memory that was used for image processing. Unfortunately, due to the long-lived object from step (2), this would be in the middle of the sbrk()ed memory, and so malloc would have to hold onto the memory for possible future use, unable to return it to the kernel. The right solution to the problem is to get rid of the hard line between data segment and heap, and use the address space that is currently reserved for sbrk() as the last resort for mmap() requests, thus allowing the data segment and heap to gracefully grow toward each other until memory is exhausted. We really don't need the data segment anymore except for legacy support of applications that do their own memory management via sbrk(). However, when I talked with the VM wizards, I was sufficiently intimidated by the difficulty of removing the hard-coded data segment size that I did not pursue it any further. Personally, in the absence of a dynamic boundary between the data segment and the heap, I would be quite happy to completely disable sbrk() support in jemalloc, and let those who really need that last 512 MB of address space adjust resource limits for their applications as necessary. In practice, I expect this would cause people far less trouble than does the current state of affairs. Jason