From owner-freebsd-ports@FreeBSD.ORG Sun Sep 7 23:42:56 2003 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 05F7C16A4BF; Sun, 7 Sep 2003 23:42:56 -0700 (PDT) Received: from HAL9000.homeunix.com (12-233-57-131.client.attbi.com [12.233.57.131]) by mx1.FreeBSD.org (Postfix) with ESMTP id E935443FE1; Sun, 7 Sep 2003 23:42:54 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.9/8.12.9) with ESMTP id h886gntA007720; Sun, 7 Sep 2003 23:42:49 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.9/8.12.9/Submit) id h886gmXT007719; Sun, 7 Sep 2003 23:42:49 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Date: Sun, 7 Sep 2003 23:42:48 -0700 From: David Schultz To: Anders Nordby , Mike Harding , stable@FreeBSD.ORG, ports@FreeBSD.ORG, adrian@FreeBSD.ORG, tjr@FreeBSD.ORG, phk@FreeBSD.ORG Message-ID: <20030908064248.GA7522@HAL9000.homeunix.com> Mail-Followup-To: Anders Nordby , Mike Harding , stable@FreeBSD.org, ports@FreeBSD.org, adrian@freebsd.org, tjr@FreeBSD.org, phk@FreeBSD.org References: <20030906141108.GA13082@totem.fix.no> <20030906151719.A83B65350@netcom1.netcom.com> <20030907152051.GA44042@totem.fix.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20030907152051.GA44042@totem.fix.no> Subject: Re: Squid memory leaks in -stable using libc malloc X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2003 06:42:56 -0000 On Sun, Sep 07, 2003, Anders Nordby wrote: > Hi, > > On Sat, Sep 06, 2003 at 08:17:19AM -0700, Mike Harding wrote: > > Squid uses more memory than you assign to cache_mem, this is > > documented in the Squid FAQ, section 8. cache_mem is sort of a > > 'suggested' value, it's normal for squid to use a lot more memory than > > cache_mem. > > I know this, and I know the contents of the FAQ. However, there's a huge > difference between what top reports as memory used (SIZE) and what cache > manager reports as total allocated KB. In a running process I have just > now, top says Squid uses 1235 MB of memory, while cache manager reports > 663 MB allocated (contents of /proc//map attached). That's after > running for around 10 hours. If I use dlmalloc (I do on one system), the > numbers are 2291 MB according to top and 2221 MB according to cache > manager after running for 30 hours. Do you see the difference? To me, > there seems to be a leak, or at least very unfavourable results with > phkmalloc and Squid. phkmalloc satisfies allocations smaller than the system page size from power-of-two buckets, and larger allocations are rounded up to a multiple of the page size. While simple, this scheme works poorly for some applications. For example, postgresql makes some allocations that are slightly larger than a power of two due to its use of inline buffer tags. If you use malloc(3)'s 'U' option in combination with ktrace(1), you can get a good idea of exactly what sizes Squid is trying to allocate. Note also that there's a difference between heap size and physical memory usage. In a demand-paged virtual memory system, you can allocate just about as much heap space as you want practically for free; you only pay for the pages you actually touch. A better metric for the memory pressure that an application is exerting on the system is its resident set size---'RES' in top(1). phkmalloc by default is not aggressive about returning memory to the system, but it usually exhibits very good behavior from the point of view of the VM system.