Date: Fri, 2 Dec 2011 23:18:47 -0500 From: Jason Hellenthal <jhell@DataIX.net> To: Steven Hartland <killing@multiplay.co.uk> Cc: freebsd-hackers@freebsd.org, Andriy Gapon <avg@freebsd.org> Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? Message-ID: <20111203041847.GC38979@DataIX.net> In-Reply-To: <2537A2099932494E9E7019B53EBDAA34@multiplay.co.uk> References: <547298A3C38F407887E1AAAAC487DF6D@multiplay.co.uk> <4ED8E9D0.4070006@FreeBSD.org> <2537A2099932494E9E7019B53EBDAA34@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Dec 02, 2011 at 05:13:05PM -0000, Steven Hartland wrote: > ----- Original Message ----- > From: "Andriy Gapon" <avg@FreeBSD.org> > > >> Totalling up RSS from ps axo "rss" gives a total in the region of that if > >> the vm stats are out by a factor of 4, in this case it should be: 8132557 > >> which is 7.75GB a much more realistic value. > >> > >> Am I totally missing something or is there problem here? > > > > Likely more of the former than of latter. Those virtual sizes are not > > sufficiently explained, but you have been warned that those are not physical > > sizes, so I am not sure why you try to compare the virtual figures with the > > physical figures. > > My miss-understanding was due to what "virtual" actually meant. > > > Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, > > here you immediately increased your virtual size counts by 1GB, even if you hadn't > > accessed any bytes in the file yet and so none of them were in physical memory. > > The same applies to anonymous memory. > > > > P.S. the above is reveled by a cursory look through the code (which is publicly > > available btw) :-) > > Yer I did have a dig around before posting and ended up the code for vm.vmtotal, > which is where vmstat gets its info from but that's just a summation of each object's > size from vm_object_list. Thats where I got lost without an insight into what > a vm_object.size actually represents. > > Your info about mmap'ed files helped point me in the right direction as it > identified space that shows as virtual but doesn't show in swap or real ram, > which is what I was missing. > > Given this starting point the following links provided me with addtional > information:- > http://www.freebsd.org/doc/en/books/arch-handbook/vm.html > http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html > http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/ > http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf > > I was under the incorrect impression that Virtual Memory (VM) was so named as it > was a unified physical memory and swap (virtual memory), but its not that simple, > as other items such as file-backed objects also count to this total which would > never show in physical or swap allocation of other tools such as top and swapinfo. > > So what I believe is now the big cause of virtual memory uplift vs the memory > totals shown by ps / top is that the vm totals include things like file backed > memory mapped process binaries, shared libs etc many multiple times. > > This would explain why this specific machine shows the applification more than > others here as it runs thousands of very small lightweight processes. > > Thanks for pointer Andy, I now understand a lot more about the BSD VMS :) > > What do people think about expanding that entry in the man page of vmstat to > clarify just what active "virtual pages" really means? > > Regards > Steve > Thanks for your research Steve. That makes perfect sense and additions to the documentation are surely needed.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111203041847.GC38979>