From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 10:19:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4481C106566C for ; Thu, 17 Nov 2011 10:19:01 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.westchester.pa.mail.comcast.net (qmta14.westchester.pa.mail.comcast.net [76.96.59.212]) by mx1.freebsd.org (Postfix) with ESMTP id E0ADD8FC0C for ; Thu, 17 Nov 2011 10:19:00 +0000 (UTC) Received: from omta09.westchester.pa.mail.comcast.net ([76.96.62.20]) by qmta14.westchester.pa.mail.comcast.net with comcast id yAK11h0030SCNGk5EAK1f5; Thu, 17 Nov 2011 10:19:01 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta09.westchester.pa.mail.comcast.net with comcast id yAJy1h00E1t3BNj3VAJymM; Thu, 17 Nov 2011 10:19:00 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id DAC05102C1D; Thu, 17 Nov 2011 02:18:56 -0800 (PST) Date: Thu, 17 Nov 2011 02:18:56 -0800 From: Jeremy Chadwick To: Kostik Belousov Message-ID: <20111117101856.GA39096@icarus.home.lan> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111117081210.GN50300@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, Daniil Cherednik , Doug Barton , freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:19:01 -0000 On Thu, Nov 17, 2011 at 10:12:10AM +0200, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: > > On 11/16/2011 23:49, Kostik Belousov wrote: > > > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > > >> On 11/15/2011 02:09, Jeremy Chadwick wrote: > > >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > > >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > > >>>>> On 11/14/2011 12:31, Doug Barton wrote: > > >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 > > >>>>>> in a busy web hosting environment I came across the following post: > > >>>>>> > > >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html > > >>>>>> > > >>>>>> That basically describes what we're seeing as well, including the > > >>>>>> "doesn't happen on Linux" part. > > >>>>>> > > >>>>>> Does anyone have any ideas about this? > > >>>>>> > > >>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, > > >>>>>> so it seems to be something new in 8. > > >>>>> > > >>>>> Just took a closer look at our ktrace, and actually our pattern is > > >>>>> slightly different than the one in that post. In ours the second option > > >>>>> is null, but the third is set: > > >>>>> > > >>>>> 74195 httpd 0.000017 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> > > >>>>> But repeated hundreds of times in a row. > > >>>> > > >>>> The calls cannot come from rtld, they are generated by some setjmp() > > >>>> invocation. If signal-safety is not needed, sigsetjmp() should be used > > >>>> instead. > > >>>> > > >>>> Quick grep of the apache httpd source shows a single setjmp() in their > > >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). > > >>> > > >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of > > >>> the Apache folks (not just port committers) may have some insight to > > >>> Kostik's findings. > > >> > > >> Thanks to everyone for the responses. We tried Kostik's suggestion and > > >> unfortunately it didn't reduce the number of sigprocmask() calls to a > > >> statistically significant degree. > > >> > > >> Does anyone have any other ideas on ways to debug this? We're sort of > > >> running out of things to test. :-/ > > >> > > >> Given how important (and prevalent) the Apache + FreeBSD combination is, > > >> I'm kind of disturbed that we're seeing this performance problem, and if > > >> it's something in 8.x that's also in 9.x, it would be better to fix it > > >> prior to 9.0-RELEASE. > > > > > > Since my guess appeared to be not useful, > > > > Well I wouldn't say that they weren't useful, we eliminated the obvious > > candidate. So, "not good news" certainly, but not unhelpful. :) > > > > > the way forward is to identify > > > the location of the call(s) that cause the issue. I suggest compliling > > > at least apache itself, libc, rtld and libthr (if used) with debugging > > > information. Then, attach to the running apache worker with the gdb and > Note this part. > > > > set breakpoint on sigprocmask. Several backtraces from the hit breakpoint > > > should give enough data. > > > > We tried that, and got this: > > > > Loaded symbols for /libexec/ld-elf.so.1 > > 0x28183a5d in accept () from /lib/libc.so.7 > > (gdb) b sigprocmask > > Breakpoint 1 at 0x282d8f84 > > (gdb) c > > Continuing. > > no thread to satisfy query > > 0x28183a5d in accept () from /lib/libc.so.7 > > (gdb) > It seems your libc has no debugging information. > accept() is the pure syscall wrapper, it cannot call sigprocmask. > If gdb catched the PLT trampoline instead of real accept(), we would > see the rtld frames. So install libc, libthr and rtld with debug. > > Also, having debug symbols for apache itself can be useful. I'd also like to point out that enabling debugging symbols in devel/apr1 will be greatly needed here, not just in www/apache*. I'm wondering if maybe this is some sort of pthread "thing" going on. A quick grep -r sigmask of the Apache source turns up some pthread_* bits pertaining to worker. Is Apache build using WITH_THREADS? What about devel/apr1? I don't use worker MPM on any of our boxes, we actually use ITK MPM solely because of the hosting nature of what we do. I've actually never seen worker MPM in use on any *IX machine I've been on or administrated, only prefork. The Apache documentation even mentions that "if you want stability or compatibility, prefork is the choice", while "if you want scalability, worker is a better choice"[1]. These sorts of quotes often shock me given what year it is. :-) [1]: http://httpd.apache.org/docs/2.0/mpm.html -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |