From owner-freebsd-ports@FreeBSD.ORG Sun Sep 7 12:42:34 2003 Return-Path: Delivered-To: freebsd-ports@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9E22316A4BF; Sun, 7 Sep 2003 12:42:34 -0700 (PDT) Received: from snipe.mail.pas.earthlink.net (snipe.mail.pas.earthlink.net [207.217.120.62]) by mx1.FreeBSD.org (Postfix) with ESMTP id CBA8243FF3; Sun, 7 Sep 2003 12:42:33 -0700 (PDT) (envelope-from mvh@ix.netcom.com) Received: from lsanca1-ar6-4-62-202-040.lsanca1.elnk.dsl.genuity.net ([4.62.202.40] helo=netcom1.netcom.com) by snipe.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 19w5Qe-0002uY-00; Sun, 07 Sep 2003 12:42:32 -0700 Received: from [127.0.0.1] (localhost [127.0.0.1]) by netcom1.netcom.com (Postfix) with ESMTP id 35F635346; Sun, 7 Sep 2003 12:42:31 -0700 (PDT) From: Mike Harding To: Anders Nordby In-Reply-To: <20030907152051.GA44042@totem.fix.no> References: <20030906141108.GA13082@totem.fix.no> <20030906151719.A83B65350@netcom1.netcom.com> <20030907152051.GA44042@totem.fix.no> Content-Type: text/plain Message-Id: <1062963750.56044.31.camel@netcom1.netcom.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.4 Date: Sun, 07 Sep 2003 12:42:31 -0700 Content-Transfer-Encoding: 7bit cc: ports@FreeBSD.org cc: stable@FreeBSD.org cc: adrian@freebsd.org cc: tjr@FreeBSD.org cc: phk@FreeBSD.org Subject: Re: Squid memory leaks in -stable using libc malloc X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2003 19:42:34 -0000 There can of course be a difference between what malloc allocates and what the application asked for. Various malloc implementations will do things like round allocations up to a power of 2, etc. The application is only aware of the memory it asked for - the OS knows how much memory was actually allocated. 'Wasted' memory isn't an indication of a bug, it's a characteristic of the malloc used. It's only leaking memory if the memory usage grows without bound, and I've never seen that with squid, in a few years anyway. Different malloc implementations optimize for different things - the FreeBSD malloc, looking at the man page, appears to try to minimize page faults. It also looks like it uses buckets with various powers of 2. dlmalloc looks to have a maximum 'wastage' of 15 bytes per chunk. There is a good writeup of dlmalloc here: http://g.oswego.edu/dl/html/malloc.html Sounds like you might as well use dlmalloc with squid - squid does generate a lot of memory allocations, and the characteristics of squid may function better with this malloc, which is likely why they provide it as an option. You may want to suggest to the port maintainer that dlmalloc be used as the default... - Mike H. On Sun, 2003-09-07 at 08:20, Anders Nordby wrote: > Hi, > > On Sat, Sep 06, 2003 at 08:17:19AM -0700, Mike Harding wrote: > > Squid uses more memory than you assign to cache_mem, this is > > documented in the Squid FAQ, section 8. cache_mem is sort of a > > 'suggested' value, it's normal for squid to use a lot more memory than > > cache_mem. > > I know this, and I know the contents of the FAQ. However, there's a huge > difference between what top reports as memory used (SIZE) and what cache > manager reports as total allocated KB. In a running process I have just > now, top says Squid uses 1235 MB of memory, while cache manager reports > 663 MB allocated (contents of /proc//map attached). That's after > running for around 10 hours. If I use dlmalloc (I do on one system), the > numbers are 2291 MB according to top and 2221 MB according to cache > manager after running for 30 hours. Do you see the difference? To me, > there seems to be a leak, or at least very unfavourable results with > phkmalloc and Squid. > > BTW, in the first example I gave here, I (still) use phkmalloc with H > configured in malloc.conf, although H did not seem to make a difference. > > > If you are happy with dmalloc, though, go ahead and use it, just check > > out the squid FAQ about memory usage. I don't think that there is > > anything wrong with FreeBSD's malloc, it just has different > > performance characteristics. > > I agree that there doesn't have to be something technically wrong with > (phk)malloc. However, if there is, someone may have an interest in > following this up (tjr@ seems to be - I talked to him on IRC about > this). > > Cheers,