Date: Sat, 19 Feb 2011 10:04:15 +0330 From: "Payam Poursaied" <me@payam124.com> To: "'Jeremy Chadwick'" <freebsd@jdc.parodius.com> Cc: freebsd-apache@freebsd.org Subject: RE: apache 100% cpu usage Message-ID: <008701cbcfff$0a721590$1f5640b0$@com> In-Reply-To: <20110219053032.GA76790@icarus.home.lan> References: <190201cbcfec$c76297f0$5627c7d0$@com> <20110219053032.GA76790@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
> -----Original Message-----
> From: Jeremy Chadwick [mailto:freebsd@jdc.parodius.com]
> Sent: Saturday, February 19, 2011 9:01 AM
> To: Payam Poursaied
> Cc: freebsd-apache@freebsd.org
> Subject: Re: apache 100% cpu usage
>
> On Sat, Feb 19, 2011 at 07:53:31AM +0330, Payam Poursaied wrote:
> > Hi all
> > I faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE.
> > The problem is that httpd processes frequently reach 100% CPU usage and stay
> > at this level of CPU usage.
> > This is a sample of top output.
> >
> > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> > 77051 www 1 118 0 208M 86080K CPU4 4 87.6H 100.00% httpd
> > 9403 www 1 118 0 215M 93340K CPU2 2 65.0H 100.00% httpd
> > 78430 www 1 118 0 226M 100M CPU0 0 58:57 100.00% httpd
> > 95332 www 1 50 0 205M 83624K sbwait 12 0:02 4.49% httpd
> > 89664 www 1 46 0 210M 88144K sbwait 11 0:05 2.59% httpd
> >
> > I think there might be a bug that caused something like endless loop but I
> > am newbie to gdb and debugging and I could not find any relevant point to
> > get into it.
> > I tried to compile apache in debug mode and then wait to see a suspicious
> > process and then attached GDB66 to that cpu intensive httpd process to find
> > out what is going on. It seems the problem is related to something in
> > /lib/libc.so.7 which I could not find out how to compile it in debug mode.
> >
> > The result of "list" and "where" command of GDB are as below, I appreciate
> > if you could point me to right direction to examine this problem deeply.
> > As an extra information, mod_perl is heavily used trough this apache
> >
> > (gdb) l
> > 5640
> > 5641 #ifdef TPF
> > 5642 EBW_AREA input_parms;
> > 5643 ecbptr()->ebrout = PRIMECRAS;
> > 5644 input_parms = * (EBW_AREA *)(&(ecbptr()->ebw000));
> > 5645 #endif
> > 5646
> > 5647 MONCONTROL(0);
> > 5648
> > 5649 common_init();
> > (gdb) w
> > Ambiguous command "w": watch, wh, whatis, where, while, while-stepping,
> > winheight, ws.
> > (gdb) where
> > #0 0x000000080093a9f5 in ?? () from /lib/libc.so.7
> > #1 0x000000080093d505 in ?? () from /lib/libc.so.7
> > #2 0x000000080093d702 in ?? () from /lib/libc.so.7
> > #3 0x000000080094352e in ?? () from /lib/libc.so.7
> > #4 0x0000000800944924 in realloc () from /lib/libc.so.7
> > #5 0x000000080307a50f in Perl_safesysrealloc () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #6 0x00000008030a6156 in Perl_sv_grow () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #7 0x000000080309e32e in Perl_sv_vcatpvfn () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #8 0x00000008030a7b42 in Perl_vnewSVpvf () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #9 0x00000008030a7bf8 in Perl_newSVpvf () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #10 0x000000080303da32 in Perl_gv_fetchmethod_autoload () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #11 0x000000080308edca in S_method_common () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #12 0x000000080308f462 in Perl_pp_method () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #13 0x000000080308eb9e in Perl_runops_standard () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #14 0x0000000803036191 in Perl_call_sv () from
> > /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so
> > #15 0x0000000802eabe36 in perl_call_handler () from
> > /usr/local/libexec/apache/libperl.so
> > #16 0x0000000802eac87a in perl_run_stacked_handlers () from
> > /usr/local/libexec/apache/libperl.so
> > #17 0x0000000802eae2b0 in perl_handler () from
> > /usr/local/libexec/apache/libperl.so
> > #18 0x000000000040ee42 in ap_invoke_handler (r=0x804624060) at
> > http_config.c:476
> > #19 0x0000000000427849 in process_request_internal (r=0x804624060) at
> > http_request.c:1299
> > #20 0x00000000004278a4 in ap_process_request (r=0x804624060) at
> > http_request.c:1315
> > #21 0x000000000041d423 in child_main (child_num_arg=14) at http_main.c:4885
> > #22 0x000000000041d734 in make_child (s=0x800c05060, slot=14,
> > now=1297853738) at http_main.c:5055
> > #23 0x000000000041db6b in perform_idle_server_maintenance () at
> > http_main.c:5256
> > #24 0x000000000041e159 in standalone_main (argc=1, argv=0x7fffffffebd8) at
> > http_main.c:5520
> > #25 0x000000000041e7a4 in main (argc=1, argv=0x7fffffffebd8) at
> > http_main.c:5773
>
> This looks like you have a mod_perl problem, or a script that's doing
> it. It's probably not an Apache issue.
>
> ktrace would be of more help here than gdb. It will at least show you
> what the children are doing in real-time, syscall-wise. Correlating
> this with underlying perl code is up to you/your problem, sadly.
>
> Good luck.
>
Thank you for your fast response. I used truss and ktrace, but there wasn't any output to show any system calls or ..
may I know how I can get inside and explore the problem? How it is possible to build /lib/libc.so.7 in debug mode? Or
any general how-to/instruction on debugging is appreciated.
Best Regards
-payam
[-- Attachment #2 --]
0 *H
01 0 + 0 *H
B0200
*H
0{1 0 UGB10U Greater Manchester10U Salford10U
Comodo CA Limited1!0 U AAA Certificate Services0
040101000000Z
281231235959Z0{1 0 UGB10U Greater Manchester10U Salford10U
Comodo CA Limited1!0 U AAA Certificate Services0"0
*H
0
@nv MEDFȃ* ]P1 p"I-Tc̶nhF SL$rNT
z3`ډU"XOhF'v5,^deHav PfxbV1 8'2Xok+c_s8x6Qx:B /I-߬tMG)b&{>%ݝ5h Ä
^/ 00 U
#>)00U 0U 00{U t0r08642http://crl.comodoca.com/AAACertificateServices.crl06420http://crl.comodo.net/AAACertificateServices.crl0
*H
V{DO X̦Ihv]`֍PO&N氥tTAOZ``J¿Ĺt-}kF/j4,}Z
/\:l7U S@lXen<ZƞYH0!el!s7Χ,,&"`^tԢShnlhV+8: k-?cb,j AP96n00Šq_Mtq40
*H
0{1 0 UGB10U Greater Manchester10U Salford10U
Comodo CA Limited1!0 U AAA Certificate Services0
040101000000Z
281231235959Z01 0 UUS1 0 UUT10USalt Lake City1 0 U
The USERTRUST Network1!0 U http://www.usertrust.com1604U-UTN-USERFirst-Client Authentication and Email0"0
*H
0
9}A;bF7`u9eJGHjM5BI/ |1Nd.)բdąQ5yNh{ zɤ2 O0 nFxoY^/m/묡j.g5y iF͠ v:z'[=s "HaLi.1 ,CZ qYں
gT:
we tbh ~GeMW(t40b0, '0#0 U #0
#>)00 U g}ĝ&p KPH|=n}0U 0U 00 U %0++0U
00U 0{U t0r08642http://crl.comodoca.com/AAACertificateServices.crl06420http://crl.comodo.net/AAACertificateServices.crl0 `HB0
*H
<~ v9<O ૄ]Te;m |7,%T_!7OT klE`-QLf<J?VvÂOl atG@We"'gOWdZٍ /i)J /LQFĊ7N 1hǞċ~2hD*Q`Mt:C29V:RAC3'9N&9≸])&A곛wuʵeJc>D^s 0'0|
;a^?Uh0
*H
01 0 UUS1 0 UUT10USalt Lake City1 0 U
The USERTRUST Network1!0 U http://www.usertrust.com1604U-UTN-USERFirst-Client Authentication and Email0
100404000000Z
110404235959Z01503U ,Comodo Trust Network - PERSONA NOT VALIDATED1F0DU =Terms and Conditions of use: http://www.comodo.net/repository1 0 U (c)2003 Comodo Limited10UPayam Poursaied1 0 *H
me@payam124.com0"0
*H
0
oi'D!ƽx ɲFmK=bLCrNoiI>Z+g6ԑۯ "\ӕk QC:0wƢl gx|'nY'B)"լ^gh/W~dAI k`x%CPuc,Gb!-CXX{|p@dmD2ρj㋬>63U fioCdJр#ڈdcE3n"""aI
QPRccyAzѤ (M |l"K<ʫuu9L,
:+UAe Qod{' Yi -{\E<į?Z2_ i UҖ#2
`ݠLSUU ;GFV3 Ѩt%q|j ^ s\fwXSyc)XŶ&pcOü\]if(h_[z/(J\xd4Լlz
w M|T 0
0 U #0g}ĝ&p KPH|=n}0 U OTI
/z@$z3`0U 0 U 0 0 U %0+ +10 `HB 0FU ?0=0; +10+0)+ https://secure.comodo.net/CPS0U 00LJHFhttp://crl.comodoca.com/UTN-USERFirst-ClientAuthenticationandEmail.crl0JHFDhttp://crl.comodo.net/UTN-USERFirst-ClientAuthenticationandEmail.crl0l+`0^06+0*http://crt.comodoca.com/UTNAAAClientCA.crt0$+0http://ocsp.comodoca.com0U 0me@payam124.com0
*H
HZ `3Y]4$B:~Fndeʪ
{Q 4R}6c芳 J4B8r|7l2wrI2U=L
ڠpR&I6q(1j}2&bJXv,^ A
?!.
mD] `v!]GgI&}X^e
Cw הKZUc B?y"$Ze%U"72?t~mVY}te10001 0 UUS1 0 UUT10USalt Lake City1 0 U
The USERTRUST Network1!0 U http://www.usertrust.com1604U-UTN-USERFirst-Client Authentication and Email|
;a^?Uh0 + 0 *H
1 *H
0 *H
1
110219063411Z0# *H
1^zXZ!Q!ֳ0 *H
100 `He*0 `He0
*H
0 `He0*H
0+0
*H
@0
*H
(0+0 `He0 `He0 `He0
*H
0 +71001 0 UUS1 0 UUT10USalt Lake City1 0 U
The USERTRUST Network1!0 U http://www.usertrust.com1604U-UTN-USERFirst-Client Authentication and Email|
;a^?Uh0 *H
1Ơ01 0 UUS1 0 UUT10USalt Lake City1 0 U
The USERTRUST Network1!0 U http://www.usertrust.com1604U-UTN-USERFirst-Client Authentication and Email|
;a^?Uh0
*H
Az,@:2?%-ce< C^ERo99 @~՚Eٱ ƒBGՌ3nV؟@`
%j:t5۸rzau/[=L4C7MK
={< q& ~ƍ )@lU
~g`۾eQYS<TWowZw^#^&zO\!Ě >=?0舴'S)i6Jp܀jJwdW..~p m MO@ P Y*cI~i<q̶
G.`-C!0 hL ~ߌթPM5!UDwa n̋ XbG(hvY?Xܞ̥Z]
ڪHե 7su}٤] g4yEmJ $H
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?008701cbcfff$0a721590$1f5640b0$>