Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 28 Oct 2008 15:34:33 -0400
From:      =?ISO-8859-1?Q?Francis_Dub=E9?= <freebsd@optiksecurite.com>
To:        Chuck Swiger <cswiger@mac.com>
Cc:        Jeremy Chadwick <koitsu@FreeBSD.org>, freebsd-questions@freebsd.org
Subject:   Re: collecting pv entries -- suggest increasing PMAP_SHPGPERPROC
Message-ID:  <49076949.3000207@optiksecurite.com>
In-Reply-To: <25FDD059-6BE7-4FAB-8FEB-C3D88789EFF2@mac.com>
References:  <49060AE0.3000301@optiksecurite.com> <E0956D53-6EA3-4E70-91B2-6606EA48869A@mac.com> <49061898.60903@optiksecurite.com> <6DE465B2-C9D2-4A97-994F-D08CD7FE6211@mac.com> <20081027200253.GA29814@icarus.home.lan> <49072561.90904@optiksecurite.com> <20081028162511.GA53758@icarus.home.lan> <4907428A.4030305@optiksecurite.com> <25FDD059-6BE7-4FAB-8FEB-C3D88789EFF2@mac.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Chuck Swiger a écrit :
> On Oct 28, 2008, at 9:49 AM, Francis Dubé wrote:
>>>> Here's an example of top's output regarding our httpd process :
>>>> 54326 apache        1  96    0   156M 13108K select 1   0:00  0.15% 
>>>> httpd
>>>> 54952 apache        1  96    0   156M 12684K select 1   0:00  0.10% 
>>>> httpd
>>>> 52343 apache        1   4    0   155M 12280K select 0   0:01  0.10% 
>>>> httpd
>>>>
>>>> Most of our page are in HTML with a LOT of images. Few PHP pages, 
>>>> very light PHP processing.
>>>>
>>>> 156M x 450 process = way more RAM than what we have (same for 
>>>> RES).  Concretely, how must I interpret these results?
>
> First, your Apache children are huge, at least for FreeBSD.  :-)  
> Also, they are mostly paged out, which suggests your system is under 
> significant VM pressure, but the vmstat output would be helpful to 
> confirm.
I'll try to remove some useless modules.

Here's the output of vmstat, i should've pasted it in the previous 
message, sorry about that.

vmstat -s
3777275311 cpu context switches
2105577673 device interrupts
359873900 software interrupts
2670696893 traps
2635245695 system calls
       43 kernel threads created
 42717777  fork() calls
   824925 vfork() calls
        0 rfork() calls
     5130 swap pager pageins
     7513 swap pager pages paged in
     5266 swap pager pageouts
    10722 swap pager pages paged out
   518980 vnode pager pageins
  1659001 vnode pager pages paged in
  5717865 vnode pager pageouts
 11193440 vnode pager pages paged out
     5530 page daemon wakeups
140578661 pages examined by the page daemon
  1701262 pages reactivated
2968698933 copy-on-write faults
  3856240 copy-on-write optimized faults
4090371353 zero fill pages zeroed
3851399420 zero fill pages prezeroed
   457318 intransit blocking page faults
1628587285 total VM faults taken
        0 pages affected by kernel thread creation
3234876655 pages affected by  fork()
 88075637 pages affected by vfork()
        0 pages affected by rfork()
1591911567 pages freed
        2 pages freed by daemon
4139768534 pages freed by exiting processes
   331854 pages active
   367993 pages inactive
    41103 pages in VM cache
   118472 pages wired down
    93794 pages free
     4096 bytes per page
125580578352 total name lookups
          cache hits (98% pos + 0% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%
>
>>> It's as I expected -- you don't understand the difference between
>>> SIZE (SZ) and RES (RSS).  The simple version:
>>>
>>> SIZE == amount of memory that's shared across all processes on the
>>> machine, e.g. shared libraries.  It doesn't mean "156MB is being taken
>>> up per process".
>
> SIZE == the amount of VM address space allocated by the process.
>
> It includes things shared (copy-on-write) between many processes like 
> the shared libraries; it also includes memory-mapped files (including 
> .so's like apache modules being loaded into the process), VM allocated 
> but not yet used by malloc()/brk(), the stack, and so forth.
>
>>> RES == amount of memory that's specifically allocated to that 
>>> individual
>>> process.  The three httpd processes above are taking up a total of
>>> ~38MBytes of memory (13108K + 12684K + 12280K).
>
> RES == the amount of process VM that is resident in actual physical 
> RAM; the rest of the process is paged out to the swapfile or 
> filesystem for memory-mapped files.
>
>> As I said, even with RES the numbers dont seems to have any sense.
>>
>> Let's say 12500K x 450 = ~5500MBytes. Considering there's a lot of 
>> process other than Apache running on the server...there's something 
>> wrong. Is there something shared in RES too ?
>
> Yep.  Quite probably a lot, but the amount of memory which is specific 
> to just that process is not easily found from FreeBSD's top, regrettably.
>
> For the sake of example, and because the same explanation applies 
> pretty closly to FreeBSD, consider an httpd running on a MacOSX 
> system.  Here's top output, which includes columns "RPRVT" for 
> "resident memory used by just this process", "RSHRD" which is 
> "resident, shared with other processes", "RSIZE" which is FreeBSD's 
> "RES", and "VSIZE", which is FreeBSD's "SIZE":
>
> Processes:  136 total, 4 running, 132 sleeping... 215 threads        
> 11:06:40
> Load Avg:  1.71, 1.66, 1.62     CPU usage:  12.5% user, 59.7% sys, 
> 27.8% idle
> SharedLibs: num =  141, resident = 18.3M code, 2.92M data, 6.40M LinkEdit
> MemRegions: num = 10360, resident =  101M + 5.91M private,  159M shared
> PhysMem:   159M wired,  252M active, 99.0M inactive,  510M used, 1.50G 
> free
> VM: 7.16G + 88.8M   1378510(0) pageins, 88743(0) pageouts
>
>   PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  
> VSIZE
>  2868 httpd        0.0% 43:21.28   1    12    92  1.82M   144M  
> 72.9M   169M
>  2869 httpd        0.0% 46:29.45   1    12    92  1.95M   144M  
> 73.2M   169M
>  2870 httpd        0.0% 46:55.84   1    12    92  1.89M   144M  
> 73.0M   169M
>
> ...and the vmmap command, documented here:
>
>   
> http://developer.apple.com/documentation/Darwin/Reference/ManPages/man1/vmmap.1.html 
>
>
> ...provides detailed info about a single process' VM usage:
>
> # vmmap 2870
> Virtual Memory Map of process 2870 (httpd)
> Output report format:  2.0
>
> ==== Non-writable regions for process 2870
> __PAGEZERO             00000000-00001000 [    4K] ---/--- SM=NUL  
> /usr/sbin/httpd
> __TEXT                 00001000-00050000 [  316K] r-x/rwx SM=COW  
> /usr/sbin/httpd
> __LINKEDIT             0005a000-00065000 [   44K] r--/rwx SM=COW  
> /usr/sbin/httpd
> __TEXT                 00065000-00068000 [   12K] r-x/rwx SM=COW  
> /usr/libexec/httpd/mod_log_config.so
> __LINKEDIT             00069000-0006a000 [    4K] r--/rwx SM=COW  
> /usr/libexec/httpd/mod_log_config.so
> __TEXT                 0006a000-0006c000 [    8K] r-x/rwx SM=COW  
> /usr/libexec/httpd/mod_mime.so
> __LINKEDIT             0006d000-0006e000 [    4K] r--/rwx SM=COW  
> /usr/libexec/httpd/mod_mime.so
> [ ... ]
> __DATA                 a1a0e000-a1a20000 [   72K] r--/r-- SM=COW  
> /usr/lib/libcrypto.0.9.7.dylib
> __DATA                 a1a20000-a1a23000 [   12K] r--/r-- SM=COW  
> /usr/lib/libcrypto.0.9.7.dylib
> __DATA                 a4f2c000-a4f2f000 [   12K] r--/r-- SM=COW  
> /usr/lib/libssl.0.9.7.dylib
> __DATA                 a7233000-a7235000 [    8K] r--/r-- SM=NUL  
> /System/Library/Perl/lib/5.8/libperl.dylib
> system                 fffec000-fffef000 [   12K] ---/rwx SM=NUL  
> commpage [libobjc.A.dylib]
> system                 fffef000-ffff0000 [    4K] r-x/rwx SM=COW  
> commpage [libobjc.A.dylib]
> system                 ffff8000-ffffa000 [    8K] r--/r-- SM=SHM  
> commpage [libSystem.B.dylib]
>
> ==== Writable regions for process 2870
> __DATA                 00050000-00059000 [   36K] rw-/rwx SM=COW  
> /usr/sbin/httpd
> __DATA                 00059000-0005a000 [    4K] rw-/rwx SM=COW  
> /usr/sbin/httpd
> __DATA                 00068000-00069000 [    4K] rw-/rwx SM=COW  
> /usr/libexec/httpd/mod_log_config.so
> __DATA                 0006c000-0006d000 [    4K] rw-/rwx SM=COW  
> /usr/libexec/httpd/mod_mime.so
> [ ... ]
> __DATA                 a0a3a000-a0a4f000 [   84K] rw-/rw- SM=COW  
> /usr/lib/libobjc.A.dylib
> __OBJC                 a0a4f000-a0a50000 [    4K] rw-/rw- SM=COW  
> /usr/lib/libobjc.A.dylib
> __DATA                 a0b70000-a0b71000 [    4K] rw-/rw- SM=COW  
> /usr/lib/libauto.dylib
> __DATA                 a1425000-a1426000 [    4K] rw-/rw- SM=COW  
> /usr/lib/libgcc_s.1.dylib
> __DATA                 a7229000-a7233000 [   40K] rw-/rw- SM=COW  
> /System/Library/Perl/lib/5.8/libperl.dylib
> Stack                  bf800000-c0000000 [ 8192K] rw-/rwx SM=COW  
> thread 0
>
> ==== Legend
> SM=sharing mode:
>     COW=copy_on_write PRV=private NUL=empty ALI=aliased
>     SHM=shared ZER=zero_filled S/A=shared_alias
>
> ==== Summary for process 2870
> ReadOnly portion of Libraries: Total=18840KB resident=10976KB(58%) 
> swapped_out_or_unallocated=7864KB(42%)
> Writable regions: Total=51100KB written=548KB(1%) 
> resident=27276KB(53%) swapped_out=0KB(0%) unallocated=23824KB(47%)
>
> REGION TYPE             [ VIRTUAL]
> ===========             [ =======]
> MALLOC                  [  42400K]
> Stack                   [   8192K]
> VM_ALLOCATE ?           [    364K]
> __DATA                  [   1048K]
> __LINKEDIT              [   4352K]
> __OBJC                  [      4K]
> __PAGEZERO              [      4K]
> __TEXT                  [  14488K]
> mapped file             [ 120368K]
> shared memory           [      4K]
> system                  [     24K]
>
> Note that you can obtain somewhat similar information under FreeBSD 
> using the sysutils/pmap port:
>
>   PID USERNAME       THR PRI NICE   SIZE    RES STATE    TIME   WCPU 
> COMMAND
> 40037 www              1   4    0 28820K 11532K accept   1:10  0.00% 
> httpd
>
> # pmap 40037
> 40037:    /usr/local/sbin/httpd
> Address   Kbytes     RSS  Shared    Priv Mode  Mapped File
> 08048000     280     208     280       - r-x   /usr/local/sbin/httpd
> 0808E000       8       8       -       8 rw-   /usr/local/sbin/httpd
> 08090000      16       -       -       - rw-   [swap pager]
> 08094000     976       -       -       - rwx   [swap pager]
> 08188000    2420       -       -       - rwx   [swap pager]
> 083E5000    6968       -       -       - rwx   [swap pager]
> 2808E000     144      92     144       - r-x   /libexec/ld-elf.so.1
> 280B2000       8       4       8       - rw-   /libexec/ld-elf.so.1
> 280B4000      20       -       -       - rw-   [swap pager]
> 280B9000      32       -       -       - rwx   [swap pager]
> 280C1000      64      16      64       - r-x   /lib/libz.so.3
> [ ... ]
> 2920F000       4       0       4       - r-x   
> /usr/local/lib/php/20060613/dom.so
> 29210000      16       0      16       - rwx   
> /usr/local/lib/php/20060613/dom.so
> 29214000      60       -       -       - rwx   [swap pager]
> 29225000      16       -       -       - rwx   [swap pager]
>
> BFBE0000     128       -       -       - rwx   [swap pager]
> -------- ------- ------- ------- -------
> Total Kb   28740    3996   17384     456
>
> ...and the pmap manpage talks about how to understand this:
>
> "     To calculate the amount of memory a group of processes is using:
>
>            -   for each process of the same binary, count the private 
> memory
>                used for n-1 processes.
>
>            -   for each binary, count the total resident size once.
>
>      For example, using the pmap data above, we can calculate that 20 
> bash
>      binaries would used
>
>            452Kbytes * 19 + 1416Kbytes = 8588 + 1416 Kbytes
>                                        = 8588Kbytes"
>
> Regards,
Thanks for the explanation and for the tool, i've installed it. Now my 
numbers makes more sense :)



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?49076949.3000207>