Date: Sun, 7 Sep 2003 23:42:48 -0700 From: David Schultz <das@FreeBSD.ORG> To: Anders Nordby <anders@fix.no>, Mike Harding <mvh@ix.netcom.com>, stable@FreeBSD.ORG, ports@FreeBSD.ORG, adrian@FreeBSD.ORG, tjr@FreeBSD.ORG, phk@FreeBSD.ORG Subject: Re: Squid memory leaks in -stable using libc malloc Message-ID: <20030908064248.GA7522@HAL9000.homeunix.com> In-Reply-To: <20030907152051.GA44042@totem.fix.no> References: <20030906141108.GA13082@totem.fix.no> <20030906151719.A83B65350@netcom1.netcom.com> <20030907152051.GA44042@totem.fix.no>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Sep 07, 2003, Anders Nordby wrote: > Hi, > > On Sat, Sep 06, 2003 at 08:17:19AM -0700, Mike Harding wrote: > > Squid uses more memory than you assign to cache_mem, this is > > documented in the Squid FAQ, section 8. cache_mem is sort of a > > 'suggested' value, it's normal for squid to use a lot more memory than > > cache_mem. > > I know this, and I know the contents of the FAQ. However, there's a huge > difference between what top reports as memory used (SIZE) and what cache > manager reports as total allocated KB. In a running process I have just > now, top says Squid uses 1235 MB of memory, while cache manager reports > 663 MB allocated (contents of /proc/<pid>/map attached). That's after > running for around 10 hours. If I use dlmalloc (I do on one system), the > numbers are 2291 MB according to top and 2221 MB according to cache > manager after running for 30 hours. Do you see the difference? To me, > there seems to be a leak, or at least very unfavourable results with > phkmalloc and Squid. phkmalloc satisfies allocations smaller than the system page size from power-of-two buckets, and larger allocations are rounded up to a multiple of the page size. While simple, this scheme works poorly for some applications. For example, postgresql makes some allocations that are slightly larger than a power of two due to its use of inline buffer tags. If you use malloc(3)'s 'U' option in combination with ktrace(1), you can get a good idea of exactly what sizes Squid is trying to allocate. Note also that there's a difference between heap size and physical memory usage. In a demand-paged virtual memory system, you can allocate just about as much heap space as you want practically for free; you only pay for the pages you actually touch. A better metric for the memory pressure that an application is exerting on the system is its resident set size---'RES' in top(1). phkmalloc by default is not aggressive about returning memory to the system, but it usually exhibits very good behavior from the point of view of the VM system.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030908064248.GA7522>