Date: Tue, 28 Oct 2008 15:34:33 -0400 From: =?ISO-8859-1?Q?Francis_Dub=E9?= <freebsd@optiksecurite.com> To: Chuck Swiger <cswiger@mac.com> Cc: Jeremy Chadwick <koitsu@FreeBSD.org>, freebsd-questions@freebsd.org Subject: Re: collecting pv entries -- suggest increasing PMAP_SHPGPERPROC Message-ID: <49076949.3000207@optiksecurite.com> In-Reply-To: <25FDD059-6BE7-4FAB-8FEB-C3D88789EFF2@mac.com> References: <49060AE0.3000301@optiksecurite.com> <E0956D53-6EA3-4E70-91B2-6606EA48869A@mac.com> <49061898.60903@optiksecurite.com> <6DE465B2-C9D2-4A97-994F-D08CD7FE6211@mac.com> <20081027200253.GA29814@icarus.home.lan> <49072561.90904@optiksecurite.com> <20081028162511.GA53758@icarus.home.lan> <4907428A.4030305@optiksecurite.com> <25FDD059-6BE7-4FAB-8FEB-C3D88789EFF2@mac.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Chuck Swiger a écrit :
> On Oct 28, 2008, at 9:49 AM, Francis Dubé wrote:
>>>> Here's an example of top's output regarding our httpd process :
>>>> 54326 apache 1 96 0 156M 13108K select 1 0:00 0.15%
>>>> httpd
>>>> 54952 apache 1 96 0 156M 12684K select 1 0:00 0.10%
>>>> httpd
>>>> 52343 apache 1 4 0 155M 12280K select 0 0:01 0.10%
>>>> httpd
>>>>
>>>> Most of our page are in HTML with a LOT of images. Few PHP pages,
>>>> very light PHP processing.
>>>>
>>>> 156M x 450 process = way more RAM than what we have (same for
>>>> RES). Concretely, how must I interpret these results?
>
> First, your Apache children are huge, at least for FreeBSD. :-)
> Also, they are mostly paged out, which suggests your system is under
> significant VM pressure, but the vmstat output would be helpful to
> confirm.
I'll try to remove some useless modules.
Here's the output of vmstat, i should've pasted it in the previous
message, sorry about that.
vmstat -s
3777275311 cpu context switches
2105577673 device interrupts
359873900 software interrupts
2670696893 traps
2635245695 system calls
43 kernel threads created
42717777 fork() calls
824925 vfork() calls
0 rfork() calls
5130 swap pager pageins
7513 swap pager pages paged in
5266 swap pager pageouts
10722 swap pager pages paged out
518980 vnode pager pageins
1659001 vnode pager pages paged in
5717865 vnode pager pageouts
11193440 vnode pager pages paged out
5530 page daemon wakeups
140578661 pages examined by the page daemon
1701262 pages reactivated
2968698933 copy-on-write faults
3856240 copy-on-write optimized faults
4090371353 zero fill pages zeroed
3851399420 zero fill pages prezeroed
457318 intransit blocking page faults
1628587285 total VM faults taken
0 pages affected by kernel thread creation
3234876655 pages affected by fork()
88075637 pages affected by vfork()
0 pages affected by rfork()
1591911567 pages freed
2 pages freed by daemon
4139768534 pages freed by exiting processes
331854 pages active
367993 pages inactive
41103 pages in VM cache
118472 pages wired down
93794 pages free
4096 bytes per page
125580578352 total name lookups
cache hits (98% pos + 0% neg) system 0% per-directory
deletions 0%, falsehits 0%, toolong 0%
>
>>> It's as I expected -- you don't understand the difference between
>>> SIZE (SZ) and RES (RSS). The simple version:
>>>
>>> SIZE == amount of memory that's shared across all processes on the
>>> machine, e.g. shared libraries. It doesn't mean "156MB is being taken
>>> up per process".
>
> SIZE == the amount of VM address space allocated by the process.
>
> It includes things shared (copy-on-write) between many processes like
> the shared libraries; it also includes memory-mapped files (including
> .so's like apache modules being loaded into the process), VM allocated
> but not yet used by malloc()/brk(), the stack, and so forth.
>
>>> RES == amount of memory that's specifically allocated to that
>>> individual
>>> process. The three httpd processes above are taking up a total of
>>> ~38MBytes of memory (13108K + 12684K + 12280K).
>
> RES == the amount of process VM that is resident in actual physical
> RAM; the rest of the process is paged out to the swapfile or
> filesystem for memory-mapped files.
>
>> As I said, even with RES the numbers dont seems to have any sense.
>>
>> Let's say 12500K x 450 = ~5500MBytes. Considering there's a lot of
>> process other than Apache running on the server...there's something
>> wrong. Is there something shared in RES too ?
>
> Yep. Quite probably a lot, but the amount of memory which is specific
> to just that process is not easily found from FreeBSD's top, regrettably.
>
> For the sake of example, and because the same explanation applies
> pretty closly to FreeBSD, consider an httpd running on a MacOSX
> system. Here's top output, which includes columns "RPRVT" for
> "resident memory used by just this process", "RSHRD" which is
> "resident, shared with other processes", "RSIZE" which is FreeBSD's
> "RES", and "VSIZE", which is FreeBSD's "SIZE":
>
> Processes: 136 total, 4 running, 132 sleeping... 215 threads
> 11:06:40
> Load Avg: 1.71, 1.66, 1.62 CPU usage: 12.5% user, 59.7% sys,
> 27.8% idle
> SharedLibs: num = 141, resident = 18.3M code, 2.92M data, 6.40M LinkEdit
> MemRegions: num = 10360, resident = 101M + 5.91M private, 159M shared
> PhysMem: 159M wired, 252M active, 99.0M inactive, 510M used, 1.50G
> free
> VM: 7.16G + 88.8M 1378510(0) pageins, 88743(0) pageouts
>
> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE
> VSIZE
> 2868 httpd 0.0% 43:21.28 1 12 92 1.82M 144M
> 72.9M 169M
> 2869 httpd 0.0% 46:29.45 1 12 92 1.95M 144M
> 73.2M 169M
> 2870 httpd 0.0% 46:55.84 1 12 92 1.89M 144M
> 73.0M 169M
>
> ...and the vmmap command, documented here:
>
>
> http://developer.apple.com/documentation/Darwin/Reference/ManPages/man1/vmmap.1.html
>
>
> ...provides detailed info about a single process' VM usage:
>
> # vmmap 2870
> Virtual Memory Map of process 2870 (httpd)
> Output report format: 2.0
>
> ==== Non-writable regions for process 2870
> __PAGEZERO 00000000-00001000 [ 4K] ---/--- SM=NUL
> /usr/sbin/httpd
> __TEXT 00001000-00050000 [ 316K] r-x/rwx SM=COW
> /usr/sbin/httpd
> __LINKEDIT 0005a000-00065000 [ 44K] r--/rwx SM=COW
> /usr/sbin/httpd
> __TEXT 00065000-00068000 [ 12K] r-x/rwx SM=COW
> /usr/libexec/httpd/mod_log_config.so
> __LINKEDIT 00069000-0006a000 [ 4K] r--/rwx SM=COW
> /usr/libexec/httpd/mod_log_config.so
> __TEXT 0006a000-0006c000 [ 8K] r-x/rwx SM=COW
> /usr/libexec/httpd/mod_mime.so
> __LINKEDIT 0006d000-0006e000 [ 4K] r--/rwx SM=COW
> /usr/libexec/httpd/mod_mime.so
> [ ... ]
> __DATA a1a0e000-a1a20000 [ 72K] r--/r-- SM=COW
> /usr/lib/libcrypto.0.9.7.dylib
> __DATA a1a20000-a1a23000 [ 12K] r--/r-- SM=COW
> /usr/lib/libcrypto.0.9.7.dylib
> __DATA a4f2c000-a4f2f000 [ 12K] r--/r-- SM=COW
> /usr/lib/libssl.0.9.7.dylib
> __DATA a7233000-a7235000 [ 8K] r--/r-- SM=NUL
> /System/Library/Perl/lib/5.8/libperl.dylib
> system fffec000-fffef000 [ 12K] ---/rwx SM=NUL
> commpage [libobjc.A.dylib]
> system fffef000-ffff0000 [ 4K] r-x/rwx SM=COW
> commpage [libobjc.A.dylib]
> system ffff8000-ffffa000 [ 8K] r--/r-- SM=SHM
> commpage [libSystem.B.dylib]
>
> ==== Writable regions for process 2870
> __DATA 00050000-00059000 [ 36K] rw-/rwx SM=COW
> /usr/sbin/httpd
> __DATA 00059000-0005a000 [ 4K] rw-/rwx SM=COW
> /usr/sbin/httpd
> __DATA 00068000-00069000 [ 4K] rw-/rwx SM=COW
> /usr/libexec/httpd/mod_log_config.so
> __DATA 0006c000-0006d000 [ 4K] rw-/rwx SM=COW
> /usr/libexec/httpd/mod_mime.so
> [ ... ]
> __DATA a0a3a000-a0a4f000 [ 84K] rw-/rw- SM=COW
> /usr/lib/libobjc.A.dylib
> __OBJC a0a4f000-a0a50000 [ 4K] rw-/rw- SM=COW
> /usr/lib/libobjc.A.dylib
> __DATA a0b70000-a0b71000 [ 4K] rw-/rw- SM=COW
> /usr/lib/libauto.dylib
> __DATA a1425000-a1426000 [ 4K] rw-/rw- SM=COW
> /usr/lib/libgcc_s.1.dylib
> __DATA a7229000-a7233000 [ 40K] rw-/rw- SM=COW
> /System/Library/Perl/lib/5.8/libperl.dylib
> Stack bf800000-c0000000 [ 8192K] rw-/rwx SM=COW
> thread 0
>
> ==== Legend
> SM=sharing mode:
> COW=copy_on_write PRV=private NUL=empty ALI=aliased
> SHM=shared ZER=zero_filled S/A=shared_alias
>
> ==== Summary for process 2870
> ReadOnly portion of Libraries: Total=18840KB resident=10976KB(58%)
> swapped_out_or_unallocated=7864KB(42%)
> Writable regions: Total=51100KB written=548KB(1%)
> resident=27276KB(53%) swapped_out=0KB(0%) unallocated=23824KB(47%)
>
> REGION TYPE [ VIRTUAL]
> =========== [ =======]
> MALLOC [ 42400K]
> Stack [ 8192K]
> VM_ALLOCATE ? [ 364K]
> __DATA [ 1048K]
> __LINKEDIT [ 4352K]
> __OBJC [ 4K]
> __PAGEZERO [ 4K]
> __TEXT [ 14488K]
> mapped file [ 120368K]
> shared memory [ 4K]
> system [ 24K]
>
> Note that you can obtain somewhat similar information under FreeBSD
> using the sysutils/pmap port:
>
> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU
> COMMAND
> 40037 www 1 4 0 28820K 11532K accept 1:10 0.00%
> httpd
>
> # pmap 40037
> 40037: /usr/local/sbin/httpd
> Address Kbytes RSS Shared Priv Mode Mapped File
> 08048000 280 208 280 - r-x /usr/local/sbin/httpd
> 0808E000 8 8 - 8 rw- /usr/local/sbin/httpd
> 08090000 16 - - - rw- [swap pager]
> 08094000 976 - - - rwx [swap pager]
> 08188000 2420 - - - rwx [swap pager]
> 083E5000 6968 - - - rwx [swap pager]
> 2808E000 144 92 144 - r-x /libexec/ld-elf.so.1
> 280B2000 8 4 8 - rw- /libexec/ld-elf.so.1
> 280B4000 20 - - - rw- [swap pager]
> 280B9000 32 - - - rwx [swap pager]
> 280C1000 64 16 64 - r-x /lib/libz.so.3
> [ ... ]
> 2920F000 4 0 4 - r-x
> /usr/local/lib/php/20060613/dom.so
> 29210000 16 0 16 - rwx
> /usr/local/lib/php/20060613/dom.so
> 29214000 60 - - - rwx [swap pager]
> 29225000 16 - - - rwx [swap pager]
>
> BFBE0000 128 - - - rwx [swap pager]
> -------- ------- ------- ------- -------
> Total Kb 28740 3996 17384 456
>
> ...and the pmap manpage talks about how to understand this:
>
> " To calculate the amount of memory a group of processes is using:
>
> - for each process of the same binary, count the private
> memory
> used for n-1 processes.
>
> - for each binary, count the total resident size once.
>
> For example, using the pmap data above, we can calculate that 20
> bash
> binaries would used
>
> 452Kbytes * 19 + 1416Kbytes = 8588 + 1416 Kbytes
> = 8588Kbytes"
>
> Regards,
Thanks for the explanation and for the tool, i've installed it. Now my
numbers makes more sense :)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?49076949.3000207>