Date: Tue, 06 Mar 2012 17:48:47 -0700 From: Ian Lepore <freebsd@damnhippie.dyndns.org> To: Luke Marsden <luke@hybrid-logic.co.uk> Cc: freebsd-fs@freebsd.org, team@hybrid-logic.co.uk, freebsd-stable@freebsd.org Subject: Re: FreeBSD 8.2 - active plus inactive memory leak!? Message-ID: <1331081327.32194.19.camel@revolution.hippie.lan> In-Reply-To: <1331061203.2218.38.camel@pow> References: <1331061203.2218.38.camel@pow>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 2012-03-06 at 19:13 +0000, Luke Marsden wrote:
> Hi all,
>
> I'm having some trouble with some production 8.2-RELEASE servers where
> the 'Active' and 'Inact' memory values reported by top don't seem to
> correspond with the processes which are running on the machine. I have
> two near-identical machines (with slightly different workloads); on one,
> let's call it A, active + free is small (6.5G) and on the other (B)
> active + free is large (13.6G), even though they have almost identical
> sums-of-resident memory (8.3G on A and 9.3G on B).
>
> The only difference is that A has a smaller number of quite long-running
> processes (it's hosting a small number of busy sites) and B has a larger
> number of more frequently killed/recycled processes (it's hosting a
> larger number of quiet sites, so the FastCGI processes get killed and
> restarted frequently). Notably B has many more ZFS filesystems mounted
> than A (around 4,000 versus 100). The machines are otherwise under
> similar amounts of load. I hoped that the community could please help
> me understand what's going on with respect to the worryingly large
> amount of active + free memory on B.
>
> Both machines are ZFS-on-root with FreeBSD 8.2-RELEASE with uptimes
> around 5-6 days. I have recently reduced the ARC cache on both machines
> since my previous thread [1] and Wired memory usage is now stable at 6G
> on A and 7G on B with an arc_max of 4G on both machines.
>
> Neither of the machines have any swap in use:
>
> Swap: 10G Total, 10G Free
>
> My current (probably quite simplistic) understanding of the FreeBSD
> virtual memory system is that, for each process as reported by top:
>
> * Size corresponds to the total size of all the text pages for the
> process (those belonging to code in the binary itself and linked
> libraries) plus data pages (including stack and malloc()'d but
> not-yet-written-to memory segments).
> * Resident corresponds to a subset of the pages above: those pages
> which actually occupy physical/core memory. Notably pages may
> appear in size but not appear in resident for read-only text
> pages from libraries which have not been used yet or which have
> been malloc()'d but not yet written-to.
>
> My understanding for the values for the system as a whole (at the top in
> 'top') is as follows:
>
> * Active / inactive memory is the same thing: resident memory from
> processes in use. Being in the inactive as opposed to active
> list simply indicates that the pages in question are less
> recently used and therefore more likely to get swapped out if
> the machine comes under memory pressure.
> * Wired is mostly kernel memory.
> * Cache is freed memory which the kernel has decided to keep in
> case it correspond to a useful page in future; it can be cheaply
> evicted into the free list.
> * Free memory is actually not being used for anything.
>
> It seems that pages which occur in the active + inactive lists must
> occur in the resident memory of one or more processes ("or more" since
> processes can share pages in e.g. read-only shared libs or COW forked
> address space). Conversely, if a page *does not* occur in the resident
> memory of any process, it must not occupy any space in the active +
> inactive lists.
>
> Therefore the active + inactive memory should always be less than or
> equal to the sum of the resident memory of all the processes on the
> system, right?
>
> But it's not. So, I wrote a very simple Python script to add up the
> resident memory values in the output from 'top' and, on machine A:
>
> Mem: 3388M Active, 3209M Inact, 6066M Wired, 196K Cache, 11G
> Free
> There were 246 processes totalling 8271 MB resident memory
>
> Whereas on machine B:
>
> Mem: 11G Active, 2598M Inact, 7177M Wired, 733M Cache, 1619M
> Free
> There were 441 processes totalling 9297 MB resident memory
>
> Now, on machine A:
>
> 3388M active + 3209M inactive - 8271M sum-of-resident = -1674M
>
> I can attribute this negative value to shared libraries between the
> running processes (which the sum-of-res is double-counting but active +
> inactive is not). But on machine B:
>
> 11264M active + 2598M inactive - 9297M sum-of-resident = 4565M
>
> I'm struggling to explain how, when there are only 9.2G (worst case,
> discounting shared pages) of resident processes, the system is using 11G
> + 2598M = 13.8G of memory!
>
> This "missing memory" is scary, because it seems to be increasing over
> time, and eventually when the system runs out of free memory, I'm
> certain it will crash in the same way described in my previous thread
> [1].
>
> Is my understanding of the virtual memory system badly broken - in which
> case please educate me ;-) or is there a real problem here? If so how
> can I dig deeper to help uncover/fix it?
>
> Best Regards,
> Luke Marsden
>
> [1] lists.freebsd.org/pipermail/freebsd-fs/2012-February/013775.html
> [2] https://gist.github.com/1988153
>
In my experience, the bulk of the memory in the inactive category is
cached disk blocks, at least for ufs (I think zfs does things
differently). On this desktop machine I have 12G physical and typically
have roughly 11G inactive, and I can unmount one particular filesystem
where most of my work is done and instantly I have almost no inactive
and roughly 11G free.
-- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1331081327.32194.19.camel>