From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 07:49:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82D78106564A; Thu, 17 Nov 2011 07:49:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 1DB5E8FC27; Thu, 17 Nov 2011 07:49:14 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAH7nAlE009051 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Nov 2011 09:49:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAH7nAOh011871; Thu, 17 Nov 2011 09:49:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAH7n9PC011870; Thu, 17 Nov 2011 09:49:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Nov 2011 09:49:09 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="LXnoj88cJr3+5tTl" Content-Disposition: inline In-Reply-To: <4EC4ADC3.2060604@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 07:49:15 -0000 --LXnoj88cJr3+5tTl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > On 11/15/2011 02:09, Jeremy Chadwick wrote: > > On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > >> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > >>> On 11/14/2011 12:31, Doug Barton wrote: > >>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i= 386 > >>>> in a busy web hosting environment I came across the following post: > >>>> > >>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/23= 4520.html > >>>> > >>>> That basically describes what we're seeing as well, including the > >>>> "doesn't happen on Linux" part. > >>>> > >>>> Does anyone have any ideas about this? > >>>> > >>>> With incredibly similar stuff running on 7.x we didn't see this prob= lem, > >>>> so it seems to be something new in 8. > >>> > >>> Just took a closer look at our ktrace, and actually our pattern is > >>> slightly different than the one in that post. In ours the second opti= on > >>> is null, but the third is set: > >>> > >>> 74195 httpd 0.000017 RET sigprocmask 0 > >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> 74195 httpd 0.000009 RET sigprocmask 0 > >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> 74195 httpd 0.000009 RET sigprocmask 0 > >>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> > >>> But repeated hundreds of times in a row. > >> > >> The calls cannot come from rtld, they are generated by some setjmp() > >> invocation. If signal-safety is not needed, sigsetjmp() should be used > >> instead. > >> > >> Quick grep of the apache httpd source shows a single setjmp() in their > >> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(= ?, 0). > >=20 > > I hate cross-posting, but: adding freebsd-apache@ to the list. Some of > > the Apache folks (not just port committers) may have some insight to > > Kostik's findings. >=20 > Thanks to everyone for the responses. We tried Kostik's suggestion and > unfortunately it didn't reduce the number of sigprocmask() calls to a > statistically significant degree. >=20 > Does anyone have any other ideas on ways to debug this? We're sort of > running out of things to test. :-/ >=20 > Given how important (and prevalent) the Apache + FreeBSD combination is, > I'm kind of disturbed that we're seeing this performance problem, and if > it's something in 8.x that's also in 9.x, it would be better to fix it > prior to 9.0-RELEASE. Since my guess appeared to be not useful, the way forward is to identify the location of the call(s) that cause the issue. I suggest compliling at least apache itself, libc, rtld and libthr (if used) with debugging information. Then, attach to the running apache worker with the gdb and set breakpoint on sigprocmask. Several backtraces from the hit breakpoint should give enough data. High-tech solution is to link with libunwind and add code into sigprocmask() to gather the stacks. But I expect that gdb attach is enough. --LXnoj88cJr3+5tTl Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7EvHUACgkQC3+MBN1Mb4il/ACdE7xOw5J8y45ceLUICuABv6pc 300Ani90sb6Q2xiFuU75nGw573Ic0LSC =NOqe -----END PGP SIGNATURE----- --LXnoj88cJr3+5tTl--