From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 10:57:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2387106564A; Thu, 17 Nov 2011 10:57:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 674268FC16; Thu, 17 Nov 2011 10:57:48 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAHAvib6042499 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAHAviYf012457; Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAHAviCC012456; Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Nov 2011 12:57:44 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111117105744.GS50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pSPXDt+5DZRK1gNs" Content-Disposition: inline In-Reply-To: <4EC4D359.4040406@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:57:49 -0000 --pSPXDt+5DZRK1gNs Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Nov 17, 2011 at 01:26:49AM -0800, Doug Barton wrote: > On 11/17/2011 00:12, Kostik Belousov wrote: > > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: > >> On 11/16/2011 23:49, Kostik Belousov wrote: > >>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > >>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: > >>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > >>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > >>>>>>> On 11/14/2011 12:31, Doug Barton wrote: > >>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-= p4 i386 > >>>>>>>> in a busy web hosting environment I came across the following po= st: > >>>>>>>> > >>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-Octobe= r/234520.html > >>>>>>>> > >>>>>>>> That basically describes what we're seeing as well, including the > >>>>>>>> "doesn't happen on Linux" part. > >>>>>>>> > >>>>>>>> Does anyone have any ideas about this? > >>>>>>>> > >>>>>>>> With incredibly similar stuff running on 7.x we didn't see this = problem, > >>>>>>>> so it seems to be something new in 8. > >>>>>>> > >>>>>>> Just took a closer look at our ktrace, and actually our pattern is > >>>>>>> slightly different than the one in that post. In ours the second = option > >>>>>>> is null, but the third is set: > >>>>>>> > >>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> > >>>>>>> But repeated hundreds of times in a row. > >>>>>> > >>>>>> The calls cannot come from rtld, they are generated by some setjmp= () > >>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be = used > >>>>>> instead. > >>>>>> > >>>>>> Quick grep of the apache httpd source shows a single setjmp() in t= heir > >>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigset= jmp(?, 0). > >>>>> > >>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Som= e of > >>>>> the Apache folks (not just port committers) may have some insight to > >>>>> Kostik's findings. > >>>> > >>>> Thanks to everyone for the responses. We tried Kostik's suggestion a= nd > >>>> unfortunately it didn't reduce the number of sigprocmask() calls to a > >>>> statistically significant degree. > >>>> > >>>> Does anyone have any other ideas on ways to debug this? We're sort of > >>>> running out of things to test. :-/ > >>>> > >>>> Given how important (and prevalent) the Apache + FreeBSD combination= is, > >>>> I'm kind of disturbed that we're seeing this performance problem, an= d if > >>>> it's something in 8.x that's also in 9.x, it would be better to fix = it > >>>> prior to 9.0-RELEASE. > >>> > >>> Since my guess appeared to be not useful, > >> > >> Well I wouldn't say that they weren't useful, we eliminated the obvious > >> candidate. So, "not good news" certainly, but not unhelpful. :) > >> > >>> the way forward is to identify > >>> the location of the call(s) that cause the issue. I suggest compliling > >>> at least apache itself, libc, rtld and libthr (if used) with debugging > >>> information. Then, attach to the running apache worker with the gdb a= nd > > Note this part. >=20 > Right, we attached to a worker, that's why it's in accept(). :) >=20 > > It seems your libc has no debugging information. > > accept() is the pure syscall wrapper, it cannot call sigprocmask. > > If gdb catched the PLT trampoline instead of real accept(), we would > > see the rtld frames. So install libc, libthr and rtld with debug. >=20 > It's not catching there though: >=20 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > 0x28183b2d in accept () at accept.S:3 > 3 RSYSCALL(accept) > (gdb) c > Continuing. > no thread to satisfy query > 0x28183b2d in accept () at accept.S:3 > 3 RSYSCALL(accept) > (gdb) info threads > Cannot get thread info: invalid key > (gdb) Err, the other part of my message was that you shall set the breakpoint on sigprocmask. I want to see a backtrace from the breakpoint hit. Several times. The backtrace at the attach time has no use. --pSPXDt+5DZRK1gNs Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7E6KgACgkQC3+MBN1Mb4j5mgCgvbV20mLT2co6NO3NUTQlM8Ub kOQAmwU4tRvdIjYTtMfkfVwUq63h/pLe =pZru -----END PGP SIGNATURE----- --pSPXDt+5DZRK1gNs--