Date: Sat, 19 Feb 2011 07:53:31 +0330 From: "Payam Poursaied" <me@payam124.com> To: <freebsd-apache@freebsd.org> Subject: apache 100% cpu usage Message-ID: <190201cbcfec$c76297f0$5627c7d0$@com>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. ------=_NextPart_000_18FD_01CBD00A.1872FAF0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi all I faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE. The problem is that httpd processes frequently reach 100% CPU usage and stay at this level of CPU usage. This is a sample of top output. PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 77051 www 1 118 0 208M 86080K CPU4 4 87.6H 100.00% httpd 9403 www 1 118 0 215M 93340K CPU2 2 65.0H 100.00% httpd 78430 www 1 118 0 226M 100M CPU0 0 58:57 100.00% httpd 95332 www 1 50 0 205M 83624K sbwait 12 0:02 4.49% httpd 89664 www 1 46 0 210M 88144K sbwait 11 0:05 2.59% httpd 95331 www 1 46 0 205M 83464K sbwait 12 0:01 1.66% httpd 46157 www 1 47 0 258M 134M sbwait 15 3:03 1.17% httpd 89493 www 1 44 0 212M 90152K accept 13 0:20 0.10% httpd 22801 www 1 44 0 264M 137M accept 13 5:40 0.00% httpd 25440 www 1 44 0 275M 145M accept 11 5:35 0.00% httpd 30396 www 1 44 0 262M 135M accept 14 4:47 0.00% httpd 82456 www 1 44 0 233M 109M accept 12 1:37 0.00% httpd 80407 www 1 44 0 239M 110M accept 10 0:49 0.00% httpd 80097 www 1 44 0 265M 135M accept 1 0:35 0.00% httpd 88622 www 1 46 0 216M 94588K accept 15 0:16 0.00% httpd 88968 www 1 44 0 208M 86316K accept 6 0:03 0.00% httpd 89686 www 1 44 0 201M 80216K accept 11 0:01 0.00% httpd 95330 www 1 44 0 201M 77088K sbwait 10 0:00 0.00% httpd I think there might be a bug that caused something like endless loop but I am newbie to gdb and debugging and I could not find any relevant point to get into it. I tried to compile apache in debug mode and then wait to see a suspicious process and then attached GDB66 to that cpu intensive httpd process to find out what is going on. It seems the problem is related to something in /lib/libc.so.7 which I could not find out how to compile it in debug mode. The result of "list" and "where" command of GDB are as below, I appreciate if you could point me to right direction to examine this problem deeply. As an extra information, mod_perl is heavily used trough this apache (gdb) l 5640 5641 #ifdef TPF 5642 EBW_AREA input_parms; 5643 ecbptr()->ebrout = PRIMECRAS; 5644 input_parms = * (EBW_AREA *)(&(ecbptr()->ebw000)); 5645 #endif 5646 5647 MONCONTROL(0); 5648 5649 common_init(); (gdb) w Ambiguous command "w": watch, wh, whatis, where, while, while-stepping, winheight, ws. (gdb) where #0 0x000000080093a9f5 in ?? () from /lib/libc.so.7 #1 0x000000080093d505 in ?? () from /lib/libc.so.7 #2 0x000000080093d702 in ?? () from /lib/libc.so.7 #3 0x000000080094352e in ?? () from /lib/libc.so.7 #4 0x0000000800944924 in realloc () from /lib/libc.so.7 #5 0x000000080307a50f in Perl_safesysrealloc () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #6 0x00000008030a6156 in Perl_sv_grow () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #7 0x000000080309e32e in Perl_sv_vcatpvfn () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #8 0x00000008030a7b42 in Perl_vnewSVpvf () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #9 0x00000008030a7bf8 in Perl_newSVpvf () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #10 0x000000080303da32 in Perl_gv_fetchmethod_autoload () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #11 0x000000080308edca in S_method_common () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #12 0x000000080308f462 in Perl_pp_method () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #13 0x000000080308eb9e in Perl_runops_standard () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #14 0x0000000803036191 in Perl_call_sv () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #15 0x0000000802eabe36 in perl_call_handler () from /usr/local/libexec/apache/libperl.so #16 0x0000000802eac87a in perl_run_stacked_handlers () from /usr/local/libexec/apache/libperl.so #17 0x0000000802eae2b0 in perl_handler () from /usr/local/libexec/apache/libperl.so #18 0x000000000040ee42 in ap_invoke_handler (r=0x804624060) at http_config.c:476 #19 0x0000000000427849 in process_request_internal (r=0x804624060) at http_request.c:1299 #20 0x00000000004278a4 in ap_process_request (r=0x804624060) at http_request.c:1315 #21 0x000000000041d423 in child_main (child_num_arg=14) at http_main.c:4885 #22 0x000000000041d734 in make_child (s=0x800c05060, slot=14, now=1297853738) at http_main.c:5055 #23 0x000000000041db6b in perform_idle_server_maintenance () at http_main.c:5256 #24 0x000000000041e159 in standalone_main (argc=1, argv=0x7fffffffebd8) at http_main.c:5520 #25 0x000000000041e7a4 in main (argc=1, argv=0x7fffffffebd8) at http_main.c:5773 ------=_NextPart_000_18FD_01CBD00A.1872FAF0 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIQQjCCBDIw ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0 ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0 ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+ QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH 7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0 YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1 OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIHJzCCBg+gAwIBAgIQ fJ4K6co7FmGJoF7AP1XRaDANBgkqhkiG9w0BAQUFADCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgT AlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3 b3JrMSEwHwYDVQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VS Rmlyc3QtQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDAeFw0xMDA0MDQwMDAwMDBaFw0x MTA0MDQyMzU5NTlaMIHaMTUwMwYDVQQLEyxDb21vZG8gVHJ1c3QgTmV0d29yayAtIFBFUlNPTkEg Tk9UIFZBTElEQVRFRDFGMEQGA1UECxM9VGVybXMgYW5kIENvbmRpdGlvbnMgb2YgdXNlOiBodHRw Oi8vd3d3LmNvbW9kby5uZXQvcmVwb3NpdG9yeTEfMB0GA1UECxMWKGMpMjAwMyBDb21vZG8gTGlt aXRlZDEYMBYGA1UEAxMPUGF5YW0gUG91cnNhaWVkMR4wHAYJKoZIhvcNAQkBFg9tZUBwYXlhbTEy NC5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDHzG9pJwFE9yGZxr14vAyiybKW Rm2V955LwsEIPWKg/+FMrkPBck5vaUk+0BHBWhMroMHUZzbUkRbbrxQJItVc5dOVpWukjJWPC/kM UbNDOjB37tnGopMIbAtneLWDfJAnFG5Z/SfJ5fSOQqaeKSLR1axeZ2gvV+1+ZEFJIGtg/7OXeO/Q ASVD8QWdUOJ1+OxjLLoBR2KEBiG+LZWYg0NYWHuBi+R8cEC48GTM7G1EMgTPgRupxWrji6w+DsQ2 A72lyzNV2gvy/2a7ab5vQ2TXSon40YAjCIkRmLeNhIzaiJeCZPNjRfPlMw5uIssixCIHYUmdAArp UVC7uMFSY+9jecnL2PLlQXro8tGkHu4oTYHVC3yYn2wiSzzKq4l1dTlMFecsp5eO2gQfDdc6Kw9V QWUMhI69UW/0CGSJpdrG1ud79bsn+bCXHxudA+P1o1lpAB+Zii17/wFcRe88xK/IP1oy0l8J4chp IP9V/9KW7iMyDWDdoEyoU1WBVcQMO7lHtaQF2UbpVjPvqP0M/tGoE6P40HSIJXF8apiKn/0DH14g 83Pcx1yCvWZ3WPBThNX3eWMpWMW2JnAOApW3Y9LQT8O8XF3ohsnpwoJpZsIoaJK7j1+WW/MVeoSk sC8oSlyCu7D6eIRkurwQNNS8ymz2eg3xd4kYzcId3X9NwnxUHQIDAQABo4ICETCCAg0wHwYDVR0j BBgwFoAUiYJnfcSdJnAAS7RQSHzePa4Ebn0wHQYDVR0OBBYEFE+jVEkNnPkveugYQP4EJM3VejNg MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEE AbIxAQMFAjARBglghkgBhvhCAQEEBAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzAp BggrBgEFBQcCARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBM oEqgSIZGaHR0cDovL2NybC5jb21vZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50 aWNhdGlvbmFuZEVtYWlsLmNybDBKoEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VS Rmlyc3QtQ2xpZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYG CCsGAQUFBzAChipodHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYI KwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAaBgNVHREEEzARgQ9tZUBwYXlhbTEy NC5jb20wDQYJKoZIhvcNAQEFBQADggEBAEhaBABgM+NZXZM0JOmtCEIF55E6B7aHfkZuZA5lD8qq DXvGUb4RHDSQUn31NmMX0PDoirPY6xIgAIZKNKZCo/AUOBVyfOQIxxI3/GwyrndySZsT7o7t5qIy mFWNPUzTCtqgBJehttrvnLH9cBL1mRlSJobsSTYVltFxtygxahqGuX0y0iaYYqZKWHbvLNde4scL QYUN9pw/vyHWjOAu2QMNbZ2/RNNd+cIM42B2jZwhgF21R/ZnSRMm8MV9+1heZQ1Ddx0Hv9eUS+aN WoRV62OwHkI/eez76g7KIq8kkQZaZQElVSICp+o3Mv367pg/dH5th/Kpu/lW4+RZq310Zb8xggW2 MIIFsgIBATCBwzCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExh a2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8v d3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1dGhlbnRp Y2F0aW9uIGFuZCBFbWFpbAIQfJ4K6co7FmGJoF7AP1XRaDAJBgUrDgMCGgUAoIICxzAYBgkqhkiG 9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xMTAyMTkwNDIzMjhaMCMGCSqGSIb3 DQEJBDEWBBQUjqorN6k5KzBr3C+mkE7KNyRQ1jCBtwYJKoZIhvcNAQkPMYGpMIGmMAsGCWCGSAFl AwQBKjALBglghkgBZQMEARYwCgYIKoZIhvcNAwcwCwYJYIZIAWUDBAECMA4GCCqGSIb3DQMCAgIA gDAHBgUrDgMCBzANBggqhkiG9w0DAgIBQDANBggqhkiG9w0DAgIBKDAHBgUrDgMCGjALBglghkgB ZQMEAgMwCwYJYIZIAWUDBAICMAsGCWCGSAFlAwQCATAKBggqhkiG9w0CBTCB1AYJKwYBBAGCNxAE MYHGMIHDMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBD aXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cu dXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRp b24gYW5kIEVtYWlsAhB8ngrpyjsWYYmgXsA/VdFoMIHWBgsqhkiG9w0BCRACCzGBxqCBwzCBrjEL MAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UE ChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5j b20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFp bAIQfJ4K6co7FmGJoF7AP1XRaDANBgkqhkiG9w0BAQEFAASCAgCC6uW119ztECdYjlDo+APnGueF nkOZZD5IxYEeb2/2a/O86abh5Tk01tTEAXDLIdtBhUZ29XkkfP6L1rgoyo45VrcT9I0IUvTmDQWd oYMVsa1mg9zCkUaqIh+mzL0EghlpLMmuUhFCZVpr4Zv5S82clwyj2KWmLYtdp0pgGjE+u+23PI/H vh1BXABfuW0R1WAB4l2S/s73JcrK/Dw2NNHqrlqe9TSP9L1UWXQAcS2nCP8zr/4BK1h9RlGQeaBy FNubAA5XTqPlWLHSHyQAkhXquRkHXW9/CfDi4ZOz+03dFUfC+MT35kt0AQHHOEe/mLrIvo9T3GmQ bg9JLHgEBU2/euKXFTdEYCsQno8yQ4eRAtnmCce6kUTsW2YCR78E7R9PA9YVn2xxfqZP2vb15BkJ 7mJTUYfr7v+Qd8tWifur0hR6jWY7OFbaDNmyXXjB8K4VGCWL9ZXPYtoE7EbFxUbPUhTnBk0gj8xk FbgjH3kI7QI8i5KPI7os5YjxliJDmLJsPPfUBVTb6jniVpbTrNvlALfGfvMElK0uOOJVtdPosSFp 5Hervwc/2oq6WSunCPfKDjppJWA5ey23hfDawdCywQno1zhGyUXo0iPgxm7AlsgBpTFrIINbveg7 7Fvgolb0A6AXECAgFy7JG0m2Idrfz9V30EVytY+BP0oWq3IcPwAAAAAAAA== ------=_NextPart_000_18FD_01CBD00A.1872FAF0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?190201cbcfec$c76297f0$5627c7d0$>