Date: Wed, 16 Nov 2011 23:59:06 -0800 From: Doug Barton <dougb@FreeBSD.org> To: Kostik Belousov <kostikbel@gmail.com> Cc: Daniil Cherednik <dcherednik@masterhost.ru>, freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask Message-ID: <4EC4BECA.5040705@FreeBSD.org> In-Reply-To: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/16/2011 23:49, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>> in a busy web hosting environment I came across the following post: >>>>>> >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>> >>>>>> That basically describes what we're seeing as well, including the >>>>>> "doesn't happen on Linux" part. >>>>>> >>>>>> Does anyone have any ideas about this? >>>>>> >>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>> so it seems to be something new in 8. >>>>> >>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>> slightly different than the one in that post. In ours the second option >>>>> is null, but the third is set: >>>>> >>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> >>>>> But repeated hundreds of times in a row. >>>> >>>> The calls cannot come from rtld, they are generated by some setjmp() >>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>> instead. >>>> >>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>> >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>> the Apache folks (not just port committers) may have some insight to >>> Kostik's findings. >> >> Thanks to everyone for the responses. We tried Kostik's suggestion and >> unfortunately it didn't reduce the number of sigprocmask() calls to a >> statistically significant degree. >> >> Does anyone have any other ideas on ways to debug this? We're sort of >> running out of things to test. :-/ >> >> Given how important (and prevalent) the Apache + FreeBSD combination is, >> I'm kind of disturbed that we're seeing this performance problem, and if >> it's something in 8.x that's also in 9.x, it would be better to fix it >> prior to 9.0-RELEASE. > > Since my guess appeared to be not useful, Well I wouldn't say that they weren't useful, we eliminated the obvious candidate. So, "not good news" certainly, but not unhelpful. :) > the way forward is to identify > the location of the call(s) that cause the issue. I suggest compliling > at least apache itself, libc, rtld and libthr (if used) with debugging > information. Then, attach to the running apache worker with the gdb and > set breakpoint on sigprocmask. Several backtraces from the hit breakpoint > should give enough data. We tried that, and got this: Loaded symbols for /libexec/ld-elf.so.1 0x28183a5d in accept () from /lib/libc.so.7 (gdb) b sigprocmask Breakpoint 1 at 0x282d8f84 (gdb) c Continuing. no thread to satisfy query 0x28183a5d in accept () from /lib/libc.so.7 (gdb) Of course I'm not the world's greatest gdb'er, so maybe there is a better way to do it? > High-tech solution is to link with libunwind and add code into sigprocmask() > to gather the stacks. But I expect that gdb attach is enough. Ok, we'll look into that, thanks. Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EC4BECA.5040705>