Date: Fri, 18 Feb 2011 21:30:32 -0800 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Payam Poursaied <me@payam124.com> Cc: freebsd-apache@freebsd.org Subject: Re: apache 100% cpu usage Message-ID: <20110219053032.GA76790@icarus.home.lan> In-Reply-To: <190201cbcfec$c76297f0$5627c7d0$@com> References: <190201cbcfec$c76297f0$5627c7d0$@com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Feb 19, 2011 at 07:53:31AM +0330, Payam Poursaied wrote: > Hi all > I faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE. > The problem is that httpd processes frequently reach 100% CPU usage and stay > at this level of CPU usage. > This is a sample of top output. > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 77051 www 1 118 0 208M 86080K CPU4 4 87.6H 100.00% httpd > 9403 www 1 118 0 215M 93340K CPU2 2 65.0H 100.00% httpd > 78430 www 1 118 0 226M 100M CPU0 0 58:57 100.00% httpd > 95332 www 1 50 0 205M 83624K sbwait 12 0:02 4.49% httpd > 89664 www 1 46 0 210M 88144K sbwait 11 0:05 2.59% httpd > 95331 www 1 46 0 205M 83464K sbwait 12 0:01 1.66% httpd > 46157 www 1 47 0 258M 134M sbwait 15 3:03 1.17% httpd > 89493 www 1 44 0 212M 90152K accept 13 0:20 0.10% httpd > 22801 www 1 44 0 264M 137M accept 13 5:40 0.00% httpd > 25440 www 1 44 0 275M 145M accept 11 5:35 0.00% httpd > 30396 www 1 44 0 262M 135M accept 14 4:47 0.00% httpd > 82456 www 1 44 0 233M 109M accept 12 1:37 0.00% httpd > 80407 www 1 44 0 239M 110M accept 10 0:49 0.00% httpd > 80097 www 1 44 0 265M 135M accept 1 0:35 0.00% httpd > 88622 www 1 46 0 216M 94588K accept 15 0:16 0.00% httpd > 88968 www 1 44 0 208M 86316K accept 6 0:03 0.00% httpd > 89686 www 1 44 0 201M 80216K accept 11 0:01 0.00% httpd > 95330 www 1 44 0 201M 77088K sbwait 10 0:00 0.00% httpd > > I think there might be a bug that caused something like endless loop but I > am newbie to gdb and debugging and I could not find any relevant point to > get into it. > I tried to compile apache in debug mode and then wait to see a suspicious > process and then attached GDB66 to that cpu intensive httpd process to find > out what is going on. It seems the problem is related to something in > /lib/libc.so.7 which I could not find out how to compile it in debug mode. > > The result of "list" and "where" command of GDB are as below, I appreciate > if you could point me to right direction to examine this problem deeply. > As an extra information, mod_perl is heavily used trough this apache > > (gdb) l > 5640 > 5641 #ifdef TPF > 5642 EBW_AREA input_parms; > 5643 ecbptr()->ebrout = PRIMECRAS; > 5644 input_parms = * (EBW_AREA *)(&(ecbptr()->ebw000)); > 5645 #endif > 5646 > 5647 MONCONTROL(0); > 5648 > 5649 common_init(); > (gdb) w > Ambiguous command "w": watch, wh, whatis, where, while, while-stepping, > winheight, ws. > (gdb) where > #0 0x000000080093a9f5 in ?? () from /lib/libc.so.7 > #1 0x000000080093d505 in ?? () from /lib/libc.so.7 > #2 0x000000080093d702 in ?? () from /lib/libc.so.7 > #3 0x000000080094352e in ?? () from /lib/libc.so.7 > #4 0x0000000800944924 in realloc () from /lib/libc.so.7 > #5 0x000000080307a50f in Perl_safesysrealloc () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #6 0x00000008030a6156 in Perl_sv_grow () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #7 0x000000080309e32e in Perl_sv_vcatpvfn () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #8 0x00000008030a7b42 in Perl_vnewSVpvf () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #9 0x00000008030a7bf8 in Perl_newSVpvf () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #10 0x000000080303da32 in Perl_gv_fetchmethod_autoload () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #11 0x000000080308edca in S_method_common () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #12 0x000000080308f462 in Perl_pp_method () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #13 0x000000080308eb9e in Perl_runops_standard () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #14 0x0000000803036191 in Perl_call_sv () from > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so > #15 0x0000000802eabe36 in perl_call_handler () from > /usr/local/libexec/apache/libperl.so > #16 0x0000000802eac87a in perl_run_stacked_handlers () from > /usr/local/libexec/apache/libperl.so > #17 0x0000000802eae2b0 in perl_handler () from > /usr/local/libexec/apache/libperl.so > #18 0x000000000040ee42 in ap_invoke_handler (r=0x804624060) at > http_config.c:476 > #19 0x0000000000427849 in process_request_internal (r=0x804624060) at > http_request.c:1299 > #20 0x00000000004278a4 in ap_process_request (r=0x804624060) at > http_request.c:1315 > #21 0x000000000041d423 in child_main (child_num_arg=14) at http_main.c:4885 > #22 0x000000000041d734 in make_child (s=0x800c05060, slot=14, > now=1297853738) at http_main.c:5055 > #23 0x000000000041db6b in perform_idle_server_maintenance () at > http_main.c:5256 > #24 0x000000000041e159 in standalone_main (argc=1, argv=0x7fffffffebd8) at > http_main.c:5520 > #25 0x000000000041e7a4 in main (argc=1, argv=0x7fffffffebd8) at > http_main.c:5773 This looks like you have a mod_perl problem, or a script that's doing it. It's probably not an Apache issue. ktrace would be of more help here than gdb. It will at least show you what the children are doing in real-time, syscall-wise. Correlating this with underlying perl code is up to you/your problem, sadly. Good luck. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110219053032.GA76790>