From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 09:27:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id F17081065670; Thu, 17 Nov 2011 09:27:15 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 110C4175F70; Thu, 17 Nov 2011 09:26:49 +0000 (UTC) Message-ID: <4EC4D359.4040406@FreeBSD.org> Date: Thu, 17 Nov 2011 01:26:49 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117081210.GN50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 09:27:16 -0000 On 11/17/2011 00:12, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: >> On 11/16/2011 23:49, Kostik Belousov wrote: >>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>>>> in a busy web hosting environment I came across the following post: >>>>>>>> >>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>>>> >>>>>>>> That basically describes what we're seeing as well, including the >>>>>>>> "doesn't happen on Linux" part. >>>>>>>> >>>>>>>> Does anyone have any ideas about this? >>>>>>>> >>>>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>>>> so it seems to be something new in 8. >>>>>>> >>>>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>>>> slightly different than the one in that post. In ours the second option >>>>>>> is null, but the third is set: >>>>>>> >>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> >>>>>>> But repeated hundreds of times in a row. >>>>>> >>>>>> The calls cannot come from rtld, they are generated by some setjmp() >>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>>>> instead. >>>>>> >>>>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>>>> >>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>>>> the Apache folks (not just port committers) may have some insight to >>>>> Kostik's findings. >>>> >>>> Thanks to everyone for the responses. We tried Kostik's suggestion and >>>> unfortunately it didn't reduce the number of sigprocmask() calls to a >>>> statistically significant degree. >>>> >>>> Does anyone have any other ideas on ways to debug this? We're sort of >>>> running out of things to test. :-/ >>>> >>>> Given how important (and prevalent) the Apache + FreeBSD combination is, >>>> I'm kind of disturbed that we're seeing this performance problem, and if >>>> it's something in 8.x that's also in 9.x, it would be better to fix it >>>> prior to 9.0-RELEASE. >>> >>> Since my guess appeared to be not useful, >> >> Well I wouldn't say that they weren't useful, we eliminated the obvious >> candidate. So, "not good news" certainly, but not unhelpful. :) >> >>> the way forward is to identify >>> the location of the call(s) that cause the issue. I suggest compliling >>> at least apache itself, libc, rtld and libthr (if used) with debugging >>> information. Then, attach to the running apache worker with the gdb and > Note this part. Right, we attached to a worker, that's why it's in accept(). :) > It seems your libc has no debugging information. > accept() is the pure syscall wrapper, it cannot call sigprocmask. > If gdb catched the PLT trampoline instead of real accept(), we would > see the rtld frames. So install libc, libthr and rtld with debug. It's not catching there though: Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) info threads Cannot get thread info: invalid key (gdb) Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/