Date: Thu, 17 Nov 2011 16:39:39 +0400 From: Daniil Cherednik <dcherednik@masterhost.ru> To: <freebsd-apache@freebsd.org> Cc: Kostik Belousov <kostikbel@gmail.com>, Doug Barton <dougb@freebsd.org>, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask Message-ID: <4EC5008B.8050603@masterhost.ru> In-Reply-To: <20111117101856.GA39096@icarus.home.lan> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <20111117101856.GA39096@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
On 17.11.2011 14:18, Jeremy Chadwick wrote: > On Thu, Nov 17, 2011 at 10:12:10AM +0200, Kostik Belousov wrote: >> On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: >>> On 11/16/2011 23:49, Kostik Belousov wrote: >>>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >>>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>>>>> in a busy web hosting environment I came across the following post: >>>>>>>>> >>>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>>>>> >>>>>>>>> That basically describes what we're seeing as well, including the >>>>>>>>> "doesn't happen on Linux" part. >>>>>>>>> >>>>>>>>> Does anyone have any ideas about this? >>>>>>>>> >>>>>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>>>>> so it seems to be something new in 8. >>>>>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>>>>> slightly different than the one in that post. In ours the second option >>>>>>>> is null, but the third is set: >>>>>>>> >>>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> >>>>>>>> But repeated hundreds of times in a row. >>>>>>> The calls cannot come from rtld, they are generated by some setjmp() >>>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>>>>> instead. >>>>>>> >>>>>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>>>>> the Apache folks (not just port committers) may have some insight to >>>>>> Kostik's findings. >>>>> Thanks to everyone for the responses. We tried Kostik's suggestion and >>>>> unfortunately it didn't reduce the number of sigprocmask() calls to a >>>>> statistically significant degree. >>>>> >>>>> Does anyone have any other ideas on ways to debug this? We're sort of >>>>> running out of things to test. :-/ >>>>> >>>>> Given how important (and prevalent) the Apache + FreeBSD combination is, >>>>> I'm kind of disturbed that we're seeing this performance problem, and if >>>>> it's something in 8.x that's also in 9.x, it would be better to fix it >>>>> prior to 9.0-RELEASE. >>>> Since my guess appeared to be not useful, >>> Well I wouldn't say that they weren't useful, we eliminated the obvious >>> candidate. So, "not good news" certainly, but not unhelpful. :) >>> >>>> the way forward is to identify >>>> the location of the call(s) that cause the issue. I suggest compliling >>>> at least apache itself, libc, rtld and libthr (if used) with debugging >>>> information. Then, attach to the running apache worker with the gdb and >> Note this part. >> >>>> set breakpoint on sigprocmask. Several backtraces from the hit breakpoint >>>> should give enough data. >>> We tried that, and got this: >>> >>> Loaded symbols for /libexec/ld-elf.so.1 >>> 0x28183a5d in accept () from /lib/libc.so.7 >>> (gdb) b sigprocmask >>> Breakpoint 1 at 0x282d8f84 >>> (gdb) c >>> Continuing. >>> no thread to satisfy query >>> 0x28183a5d in accept () from /lib/libc.so.7 >>> (gdb) >> It seems your libc has no debugging information. >> accept() is the pure syscall wrapper, it cannot call sigprocmask. >> If gdb catched the PLT trampoline instead of real accept(), we would >> see the rtld frames. So install libc, libthr and rtld with debug. >> >> Also, having debug symbols for apache itself can be useful. > I'd also like to point out that enabling debugging symbols in devel/apr1 > will be greatly needed here, not just in www/apache*. > > I'm wondering if maybe this is some sort of pthread "thing" going on. A > quick grep -r sigmask of the Apache source turns up some pthread_* bits > pertaining to worker. > > Is Apache build using WITH_THREADS? What about devel/apr1? > > I don't use worker MPM on any of our boxes, we actually use ITK MPM > solely because of the hosting nature of what we do. I've actually never > seen worker MPM in use on any *IX machine I've been on or administrated, > only prefork. The Apache documentation even mentions that "if you want > stability or compatibility, prefork is the choice", while "if you want > scalability, worker is a better choice"[1]. These sorts of quotes often > shock me given what year it is. :-) > > [1]: http://httpd.apache.org/docs/2.0/mpm.html > We use ITK MPM too, but we have big trouble with performance on FreeBSD. Also, I have to say we can`t use keep-alive connection, so apache creates new child for each request. -- С уважением, Daniil Cherednik .masterhost
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EC5008B.8050603>