Date: Sun, 07 Sep 2003 12:42:31 -0700 From: Mike Harding <mvh@ix.netcom.com> To: Anders Nordby <anders@fix.no> Cc: phk@FreeBSD.org Subject: Re: Squid memory leaks in -stable using libc malloc Message-ID: <1062963750.56044.31.camel@netcom1.netcom.com> In-Reply-To: <20030907152051.GA44042@totem.fix.no> References: <20030906141108.GA13082@totem.fix.no> <20030906151719.A83B65350@netcom1.netcom.com> <20030907152051.GA44042@totem.fix.no>
next in thread | previous in thread | raw e-mail | index | archive | help
There can of course be a difference between what malloc allocates and what the application asked for. Various malloc implementations will do things like round allocations up to a power of 2, etc. The application is only aware of the memory it asked for - the OS knows how much memory was actually allocated. 'Wasted' memory isn't an indication of a bug, it's a characteristic of the malloc used. It's only leaking memory if the memory usage grows without bound, and I've never seen that with squid, in a few years anyway. Different malloc implementations optimize for different things - the FreeBSD malloc, looking at the man page, appears to try to minimize page faults. It also looks like it uses buckets with various powers of 2. dlmalloc looks to have a maximum 'wastage' of 15 bytes per chunk. There is a good writeup of dlmalloc here: http://g.oswego.edu/dl/html/malloc.html Sounds like you might as well use dlmalloc with squid - squid does generate a lot of memory allocations, and the characteristics of squid may function better with this malloc, which is likely why they provide it as an option. You may want to suggest to the port maintainer that dlmalloc be used as the default... - Mike H. On Sun, 2003-09-07 at 08:20, Anders Nordby wrote: > Hi, > > On Sat, Sep 06, 2003 at 08:17:19AM -0700, Mike Harding wrote: > > Squid uses more memory than you assign to cache_mem, this is > > documented in the Squid FAQ, section 8. cache_mem is sort of a > > 'suggested' value, it's normal for squid to use a lot more memory than > > cache_mem. > > I know this, and I know the contents of the FAQ. However, there's a huge > difference between what top reports as memory used (SIZE) and what cache > manager reports as total allocated KB. In a running process I have just > now, top says Squid uses 1235 MB of memory, while cache manager reports > 663 MB allocated (contents of /proc/<pid>/map attached). That's after > running for around 10 hours. If I use dlmalloc (I do on one system), the > numbers are 2291 MB according to top and 2221 MB according to cache > manager after running for 30 hours. Do you see the difference? To me, > there seems to be a leak, or at least very unfavourable results with > phkmalloc and Squid. > > BTW, in the first example I gave here, I (still) use phkmalloc with H > configured in malloc.conf, although H did not seem to make a difference. > > > If you are happy with dmalloc, though, go ahead and use it, just check > > out the squid FAQ about memory usage. I don't think that there is > > anything wrong with FreeBSD's malloc, it just has different > > performance characteristics. > > I agree that there doesn't have to be something technically wrong with > (phk)malloc. However, if there is, someone may have an interest in > following this up (tjr@ seems to be - I talked to him on IRC about > this). > > Cheers,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1062963750.56044.31.camel>