From owner-freebsd-apache@FreeBSD.ORG Sun Nov 13 10:11:00 2011 Return-Path: Delivered-To: apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA035106566B for ; Sun, 13 Nov 2011 10:10:59 +0000 (UTC) (envelope-from umar_draz@yahoo.com) Received: from nm17-vm0.bullet.mail.sp2.yahoo.com (nm17-vm0.bullet.mail.sp2.yahoo.com [98.139.91.212]) by mx1.freebsd.org (Postfix) with SMTP id C3E968FC14 for ; Sun, 13 Nov 2011 10:10:59 +0000 (UTC) Received: from [98.139.91.67] by nm17.bullet.mail.sp2.yahoo.com with NNFMP; 13 Nov 2011 09:57:38 -0000 Received: from [98.139.91.27] by tm7.bullet.mail.sp2.yahoo.com with NNFMP; 13 Nov 2011 09:57:38 -0000 Received: from [127.0.0.1] by omp1027.mail.sp2.yahoo.com with NNFMP; 13 Nov 2011 09:57:38 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 438847.83855.bm@omp1027.mail.sp2.yahoo.com Received: (qmail 17694 invoked by uid 60001); 13 Nov 2011 09:57:37 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1321178257; bh=I3pnm3sdX4KYV9AHkAfuaPdOm1Ugm/H7eipJAb0mbB8=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=NkvmSOfwd/0e99hj55nFozN5ZO63H6nfi0fDIQT1zc7D+5sOFRvAjMS9Ul2kC0OruUHaVEpxBAlrQTbZ8R6Xm4MM5Ym+mQoUJwH/KYgYl7/ZnYjghhPDwWBQLcEHvIryPk3igHR6Rzyx+qYXdAocicryBPFZSPYTtbpws5nZIzg= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=BiHuWw3AmjpxI4iDeReK9ty7jfOQn1TWdYNoOqXYOcDAezdEkJbL7ynIyiuC1Zc1YFEVttAN51C8Nsn+Exx0yMWA6CXVZAA/nYy7tOevchBSO4alQNURAGsSTzU/08rH+jdrJEG5+RAFYWzcpHVp3l9YAYTmlrf2zF1fZ4qLLV8=; X-YMail-OSG: x14p5OEVM1klq.QN1wALxFH.EPTTT.BgDQKj4ohwtB.Tcvk fyfOOwxVNIXiWogpL3.OYFymB_d4iycMw4C_YAk9.TukZS_SOl0hVitn9NQe skyx73UJwSc8wyMD6WsaUmons7EsoGoNuRcnFVkFVR5Hx5IJAvmfhq9BYeYE mWRrbMrppBQOARjKMgpO.wf6yADszAhDHXaTTk64J5neKFG5pBue72Kwe5Cu S.7rgaMVzc9buliXA5YAdkeMCQV1oZspxoydEiRCopLaLIaj55x1xLPE.gle 5xP6q5_6GsNuU7xnWQGs2iX9zDl_ugjYPlxI_HlrkCN_mQuyLzo5laGNPqNC aPxDv6HRZImDjt0FpEmEXLQ66l577_uzmsNExjQRoLLu0ksNmtLTEgecox7N WVTdBcCksOjRtdmJ8KTXntWQADv4edN3buqfFldQILiAjhvgIiGaKGwbBy9y sKfY_ Received: from [182.185.202.157] by web111510.mail.gq1.yahoo.com via HTTP; Sun, 13 Nov 2011 01:57:37 PST X-Mailer: YahooMailWebService/0.8.115.325013 Message-ID: <1321178257.16765.YahooMailNeo@web111510.mail.gq1.yahoo.com> Date: Sun, 13 Nov 2011 01:57:37 -0800 (PST) From: Umar Draz To: "apache@FreeBSD.org" MIME-Version: 1.0 X-Mailman-Approved-At: Sun, 13 Nov 2011 12:06:24 +0000 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: ldap installation problem with apache X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Umar Draz List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 10:11:00 -0000 Hi, I am trying to install apache-2.2.1 on FreeBSD 8.2 64bit, I want to use ldap_authentication. Here is my make.conf WITH_LDAP_MODULES=yes But I am getting this error on make install mod_authnz_ldap.c:41:2: error: #error mod_authnz_ldap requires APR-util to have LDAP support built in. To fix add --with-ldap to ./configure. mod_authnz_ldap.c:63: error: expected specifier-qualifier-list before 'deref_options' the apr package is also installed apr-0.9.19.0.9.19_1 Apache Portability Library Please help me how to solve this error Best Regards, Umar From owner-freebsd-apache@FreeBSD.ORG Sun Nov 13 17:29:37 2011 Return-Path: Delivered-To: apache@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5BB50106564A for ; Sun, 13 Nov 2011 17:29:37 +0000 (UTC) (envelope-from ohauer@FreeBSD.org) Received: from p578be941.dip0.t-ipconnect.de (p578be941.dip0.t-ipconnect.de [87.139.233.65]) by mx1.freebsd.org (Postfix) with ESMTP id 199818FC08 for ; Sun, 13 Nov 2011 17:29:37 +0000 (UTC) Received: from [192.168.0.100] (cde1100.uni.vrs [192.168.0.100]) (Authenticated sender: ohauer) by p578be941.dip0.t-ipconnect.de (Postfix) with ESMTPSA id C0BD720624; Sun, 13 Nov 2011 18:29:31 +0100 (CET) Message-ID: <4EBFFE83.6020002@FreeBSD.org> Date: Sun, 13 Nov 2011 18:29:39 +0100 From: Olli Hauer User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Umar Draz References: <1321178257.16765.YahooMailNeo@web111510.mail.gq1.yahoo.com> In-Reply-To: <1321178257.16765.YahooMailNeo@web111510.mail.gq1.yahoo.com> X-Enigmail-Version: 1.3.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "apache@FreeBSD.org" Subject: Re: ldap installation problem with apache X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ohauer@FreeBSD.org List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 17:29:37 -0000 On 2011-11-13 10:57, Umar Draz wrote: > Hi, > > I am trying to install apache-2.2.1 on FreeBSD 8.2 64bit, I want to use ldap_authentication. Here is my make.conf > > WITH_LDAP_MODULES=yes > > But I am getting this error on make install > > mod_authnz_ldap.c:41:2: error: #error mod_authnz_ldap requires APR-util to have LDAP support built in. To fix add --with-ldap to ./configure. > mod_authnz_ldap.c:63: error: expected specifier-qualifier-list before 'deref_options' > > > the apr package is also installed > > apr-0.9.19.0.9.19_1 Apache Portability Library > > Please help me how to solve this error > Hi Umar, that's the wrong apr package! (maybe a relict from apache13) apache22 needs the apr1 port (devel/apr1) but you have apr0 installed. See ports/UPDATING entry from 20100518 $> grep -A27 20100518 /usr/ports/UPDATING $> cd ports/devel/apr1 $> make config -> add LDAP to the mix then follow the UPDATING instruction or do a manually delete of apr0 and fresh installation of devel/apr1 -- Regards, olli From owner-freebsd-apache@FreeBSD.ORG Sun Nov 13 19:02:13 2011 Return-Path: Delivered-To: apache@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42A431065675; Sun, 13 Nov 2011 19:02:12 +0000 (UTC) (envelope-from ohauer@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1B1288FC0A; Sun, 13 Nov 2011 19:02:12 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id pADJ2BCJ048094; Sun, 13 Nov 2011 19:02:11 GMT (envelope-from ohauer@freefall.freebsd.org) Received: (from ohauer@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id pADJ2BWs048089; Sun, 13 Nov 2011 19:02:11 GMT (envelope-from ohauer) Date: Sun, 13 Nov 2011 19:02:11 GMT Message-Id: <201111131902.pADJ2BWs048089@freefall.freebsd.org> To: jau@iki.fi, ohauer@FreeBSD.org, apache@FreeBSD.org, ohauer@FreeBSD.org From: ohauer@FreeBSD.org Cc: Subject: Re: ports/162305: [PATCH] www/mod_geoip2 update to 1.2.7 X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 19:02:13 -0000 Synopsis: [PATCH] www/mod_geoip2 update to 1.2.7 Responsible-Changed-From-To: apache->ohauer Responsible-Changed-By: ohauer Responsible-Changed-When: Sun Nov 13 19:01:20 UTC 2011 Responsible-Changed-Why: I'll take it http://www.freebsd.org/cgi/query-pr.cgi?pr=162305 From owner-freebsd-apache@FreeBSD.ORG Sun Nov 13 21:37:37 2011 Return-Path: Delivered-To: apache@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E9AC1065672; Sun, 13 Nov 2011 21:37:37 +0000 (UTC) (envelope-from ohauer@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 47A148FC12; Sun, 13 Nov 2011 21:37:37 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id pADLbbS8090003; Sun, 13 Nov 2011 21:37:37 GMT (envelope-from ohauer@freefall.freebsd.org) Received: (from ohauer@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id pADLbb3M089999; Sun, 13 Nov 2011 21:37:37 GMT (envelope-from ohauer) Date: Sun, 13 Nov 2011 21:37:37 GMT Message-Id: <201111132137.pADLbb3M089999@freefall.freebsd.org> To: ohauer@FreeBSD.org, apache@FreeBSD.org, ohauer@FreeBSD.org From: ohauer@FreeBSD.org Cc: Subject: Re: ports/162247: [UPDATE] devel/apr0: Update 0.9.20 X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 21:37:37 -0000 Synopsis: [UPDATE] devel/apr0: Update 0.9.20 Responsible-Changed-From-To: apache->ohauer Responsible-Changed-By: ohauer Responsible-Changed-When: Sun Nov 13 21:37:18 UTC 2011 Responsible-Changed-Why: I'll take it http://www.freebsd.org/cgi/query-pr.cgi?pr=162247 From owner-freebsd-apache@FreeBSD.ORG Mon Nov 14 02:17:55 2011 Return-Path: Delivered-To: apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 896F510656A3; Mon, 14 Nov 2011 02:17:55 +0000 (UTC) (envelope-from pgollucci@taximagic.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 399FD8FC22; Mon, 14 Nov 2011 02:17:54 +0000 (UTC) Received: by ggnk3 with SMTP id k3so8093870ggn.13 for ; Sun, 13 Nov 2011 18:17:54 -0800 (PST) Received: by 10.236.143.3 with SMTP id k3mr11174444yhj.91.1321235324176; Sun, 13 Nov 2011 17:48:44 -0800 (PST) Received: from jlhewitt.local ([204.239.250.1]) by mx.google.com with ESMTPS id 32sm56587596anu.10.2011.11.13.17.48.42 (version=SSLv3 cipher=OTHER); Sun, 13 Nov 2011 17:48:43 -0800 (PST) Message-ID: <4EC07379.9030507@p6m7g8.com> Date: Sun, 13 Nov 2011 17:48:41 -0800 From: "Philip M. Gollucci" Organization: P6M7G8 Inc. User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: ohauer@freebsd.org References: <1321178257.16765.YahooMailNeo@web111510.mail.gq1.yahoo.com> <4EBFFE83.6020002@FreeBSD.org> In-Reply-To: <4EBFFE83.6020002@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 14 Nov 2011 04:14:26 +0000 Cc: "apache@FreeBSD.org" , Umar Draz Subject: Re: ldap installation problem with apache X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 02:17:55 -0000 On 11/13/11 9:29 AM, Olli Hauer wrote: > On 2011-11-13 10:57, Umar Draz wrote: >> Hi, >> >> I am trying to install apache-2.2.1 on FreeBSD 8.2 64bit, I want to use ldap_authentication. Here is my make.conf >> >> WITH_LDAP_MODULES=yes >> >> But I am getting this error on make install >> >> mod_authnz_ldap.c:41:2: error: #error mod_authnz_ldap requires APR-util to have LDAP support built in. To fix add --with-ldap to ./configure. >> mod_authnz_ldap.c:63: error: expected specifier-qualifier-list before 'deref_options' >> >> >> the apr package is also installed >> >> apr-0.9.19.0.9.19_1 Apache Portability Library >> >> Please help me how to solve this error >> > > Hi Umar, > > that's the wrong apr package! (maybe a relict from apache13) > > apache22 needs the apr1 port (devel/apr1) but you have apr0 installed. > > See ports/UPDATING entry from 20100518 > $> grep -A27 20100518 /usr/ports/UPDATING > > $> cd ports/devel/apr1 > $> make config -> add LDAP to the mix > > then follow the UPDATING instruction or do a manually delete of apr0 and fresh installation of devel/apr1 Also 2.2.1 is anicent and CVE riddled. -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Director Operations, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From owner-freebsd-apache@FreeBSD.ORG Mon Nov 14 02:19:00 2011 Return-Path: Delivered-To: apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C1352106564A for ; Mon, 14 Nov 2011 02:19:00 +0000 (UTC) (envelope-from pgollucci@taximagic.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 88B328FC13 for ; Mon, 14 Nov 2011 02:19:00 +0000 (UTC) Received: by ywe9 with SMTP id 9so2882281ywe.13 for ; Sun, 13 Nov 2011 18:18:59 -0800 (PST) Received: by 10.236.77.232 with SMTP id d68mr11104500yhe.98.1321235263814; Sun, 13 Nov 2011 17:47:43 -0800 (PST) Received: from jlhewitt.local ([204.239.250.1]) by mx.google.com with ESMTPS id l19sm56570809anc.14.2011.11.13.17.47.42 (version=SSLv3 cipher=OTHER); Sun, 13 Nov 2011 17:47:43 -0800 (PST) Message-ID: <4EC0733D.7070208@p6m7g8.com> Date: Sun, 13 Nov 2011 17:47:41 -0800 From: "Philip M. Gollucci" Organization: P6M7G8 Inc. User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: FreeBSD developers , apache@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 14 Nov 2011 04:14:44 +0000 Cc: Subject: apache 2.0 / 2.2 X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 02:19:00 -0000 All, At AC NA 2011 -- 1) A formal statement will be coming out about 2.0.x EOL sometime in November. 2) 2.3.15 is expected to the be the last beta before 2.4.0 before the end of November. 3) That said, I think we can start adding a 24 port officially, Its not too late send fixes in either, if we have things in files/ that are not related to layout please ferry them upstream via me or the dev@httpd.a.o list, so we can easily update this new port. I'm sure the first 2-4 will be relatively fast. AFAIK, I believe ohaurer@ and maybe even clement@ have done some work for the port already, so rather then me re-inventing the wheel, can you point us back at your RIP. -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Director Operations, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. From owner-freebsd-apache@FreeBSD.ORG Mon Nov 14 11:06:15 2011 Return-Path: Delivered-To: apache@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1F35106566C for ; Mon, 14 Nov 2011 11:06:15 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C82198FC22 for ; Mon, 14 Nov 2011 11:06:15 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id pAEB6F4x082858 for ; Mon, 14 Nov 2011 11:06:15 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id pAEB6FnS082855 for apache@FreeBSD.org; Mon, 14 Nov 2011 11:06:15 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 14 Nov 2011 11:06:15 GMT Message-Id: <201111141106.pAEB6FnS082855@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: apache@FreeBSD.org Cc: Subject: Current problem reports assigned to apache@FreeBSD.org X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 11:06:16 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- f ports/162080 apache [PATCH] devel/apr1: Improved decision IPv6 o ports/161758 apache www/mod_rpaf2: slave seperation and Apache22+ limitati o ports/160702 apache devel/apr1: CLANG build produces unusable library for o ports/159812 apache [PATCH] www/apache20,www/apache22 Strip Binaries o ports/159608 apache www/apache22: apache WITH_BDB_BASE settings described o ports/158565 apache www/apache22: Add rlimits based on login class for mpm o ports/157554 apache www/apache22: Apache RLimitNPROC does not work as inte o ports/156987 apache www/apache22: Harden SSL cipher suites strength and SS o ports/156787 apache www/mod_auth_kerb2 fails on undefined symbol with base f ports/156719 apache ab: apr_socket_recv: Connection reset by peer (54) o ports/156251 apache [PATCH] Enable module by default for www/mod_fastcgi o ports/153406 apache www/apache22's SUEXEC_RSRCLIMIT option does not take e o ports/153264 apache www/apache22 and apache13-modssl -- rc.d script improv o ports/147806 apache [PATCH] www/apache20: httpd doesn't start with WITH_LD o ports/147282 apache errors when starting www/apache22 after installation o o ports/146199 apache www/apache20: port does not use make config o ports/144010 apache devel/apr1 tries to use SYSVIPC even in jails o ports/130479 apache www/apache20 and www/apache22 configure_args busted o ports/125183 apache www/apache22 wrong SUEXEC_DOCROOT o ports/124375 apache security/heimdal: www/mod_auth_kerb doesn't compile ag s ports/108169 apache www/apache20 wrong AP_SAFE_PATH for suEXEC 21 problems total. From owner-freebsd-apache@FreeBSD.ORG Tue Nov 15 00:03:39 2011 Return-Path: Delivered-To: apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B8EA106564A; Tue, 15 Nov 2011 00:03:39 +0000 (UTC) (envelope-from ohauer@FreeBSD.org) Received: from p578be941.dip0.t-ipconnect.de (p578be941.dip0.t-ipconnect.de [87.139.233.65]) by mx1.freebsd.org (Postfix) with ESMTP id EE1158FC08; Tue, 15 Nov 2011 00:03:38 +0000 (UTC) Received: from [192.168.0.100] (cde1100.uni.vrs [192.168.0.100]) (Authenticated sender: ohauer) by p578be941.dip0.t-ipconnect.de (Postfix) with ESMTPSA id 4723B2072F; Tue, 15 Nov 2011 01:03:34 +0100 (CET) Message-ID: <4EC1AC57.8070406@FreeBSD.org> Date: Tue, 15 Nov 2011 01:03:35 +0100 From: Olli Hauer User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: "Philip M. Gollucci" References: <4EC0733D.7070208@p6m7g8.com> In-Reply-To: <4EC0733D.7070208@p6m7g8.com> X-Enigmail-Version: 1.3.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: apache@freebsd.org, FreeBSD developers Subject: Re: apache 2.0 / 2.2 X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ohauer@FreeBSD.org List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 00:03:39 -0000 On 2011-11-14 02:47, Philip M. Gollucci wrote: > All, > > At AC NA 2011 -- > 1) A formal statement will be coming out about 2.0.x EOL sometime in November. > > 2) 2.3.15 is expected to the be the last beta before 2.4.0 before the end of November. > > 3) That said, I think we can start adding a 24 port officially, Its not too late send fixes in either, if we have things in files/ that are not related to layout please ferry them upstream via me or the dev@httpd.a.o list, so we can easily update this new port. I'm sure the first 2-4 will be relatively fast. AFAIK, I believe ohaurer@ and maybe even clement@ have done some work for the port already, so rather then me re-inventing the wheel, can you point us back at your RIP. > Hi Philip, until now I have no port in the pipe (out of time with other projects). -- regards, olli From owner-freebsd-apache@FreeBSD.ORG Tue Nov 15 10:22:22 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED8E3106564A for ; Tue, 15 Nov 2011 10:22:22 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.westchester.pa.mail.comcast.net (qmta14.westchester.pa.mail.comcast.net [76.96.59.212]) by mx1.freebsd.org (Postfix) with ESMTP id 997008FC0A for ; Tue, 15 Nov 2011 10:22:22 +0000 (UTC) Received: from omta24.westchester.pa.mail.comcast.net ([76.96.62.76]) by qmta14.westchester.pa.mail.comcast.net with comcast id xN8G1h00D1ei1Bg5EN977l; Tue, 15 Nov 2011 10:09:07 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta24.westchester.pa.mail.comcast.net with comcast id xN951h0151t3BNj3kN96c0; Tue, 15 Nov 2011 10:09:07 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7366E102C1D; Tue, 15 Nov 2011 02:09:04 -0800 (PST) Date: Tue, 15 Nov 2011 02:09:04 -0800 From: Jeremy Chadwick To: Kostik Belousov Message-ID: <20111115100904.GA92795@icarus.home.lan> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111115090745.GO50300@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Daniil Cherednik , Doug Barton , freebsd-stable@freebsd.org, freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 10:22:23 -0000 On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > > On 11/14/2011 12:31, Doug Barton wrote: > > > Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 > > > in a busy web hosting environment I came across the following post: > > > > > > http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html > > > > > > That basically describes what we're seeing as well, including the > > > "doesn't happen on Linux" part. > > > > > > Does anyone have any ideas about this? > > > > > > With incredibly similar stuff running on 7.x we didn't see this problem, > > > so it seems to be something new in 8. > > > > Just took a closer look at our ktrace, and actually our pattern is > > slightly different than the one in that post. In ours the second option > > is null, but the third is set: > > > > 74195 httpd 0.000017 RET sigprocmask 0 > > 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > 74195 httpd 0.000009 RET sigprocmask 0 > > 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > 74195 httpd 0.000009 RET sigprocmask 0 > > 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > > > But repeated hundreds of times in a row. > > The calls cannot come from rtld, they are generated by some setjmp() > invocation. If signal-safety is not needed, sigsetjmp() should be used > instead. > > Quick grep of the apache httpd source shows a single setjmp() in their > copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). I hate cross-posting, but: adding freebsd-apache@ to the list. Some of the Apache folks (not just port committers) may have some insight to Kostik's findings. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 06:46:28 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 1ED0F106566C; Thu, 17 Nov 2011 06:46:28 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 6EB361513C5; Thu, 17 Nov 2011 06:46:27 +0000 (UTC) Message-ID: <4EC4ADC3.2060604@FreeBSD.org> Date: Wed, 16 Nov 2011 22:46:27 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> In-Reply-To: <20111115100904.GA92795@icarus.home.lan> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Kostik Belousov , Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 06:46:28 -0000 On 11/15/2011 02:09, Jeremy Chadwick wrote: > On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>> On 11/14/2011 12:31, Doug Barton wrote: >>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>> in a busy web hosting environment I came across the following post: >>>> >>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>> >>>> That basically describes what we're seeing as well, including the >>>> "doesn't happen on Linux" part. >>>> >>>> Does anyone have any ideas about this? >>>> >>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>> so it seems to be something new in 8. >>> >>> Just took a closer look at our ktrace, and actually our pattern is >>> slightly different than the one in that post. In ours the second option >>> is null, but the third is set: >>> >>> 74195 httpd 0.000017 RET sigprocmask 0 >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>> 74195 httpd 0.000009 RET sigprocmask 0 >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>> 74195 httpd 0.000009 RET sigprocmask 0 >>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>> >>> But repeated hundreds of times in a row. >> >> The calls cannot come from rtld, they are generated by some setjmp() >> invocation. If signal-safety is not needed, sigsetjmp() should be used >> instead. >> >> Quick grep of the apache httpd source shows a single setjmp() in their >> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). > > I hate cross-posting, but: adding freebsd-apache@ to the list. Some of > the Apache folks (not just port committers) may have some insight to > Kostik's findings. Thanks to everyone for the responses. We tried Kostik's suggestion and unfortunately it didn't reduce the number of sigprocmask() calls to a statistically significant degree. Does anyone have any other ideas on ways to debug this? We're sort of running out of things to test. :-/ Given how important (and prevalent) the Apache + FreeBSD combination is, I'm kind of disturbed that we're seeing this performance problem, and if it's something in 8.x that's also in 9.x, it would be better to fix it prior to 9.0-RELEASE. Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 08:00:12 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id CAF85106564A; Thu, 17 Nov 2011 08:00:12 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 9CE8515027C; Thu, 17 Nov 2011 07:59:06 +0000 (UTC) Message-ID: <4EC4BECA.5040705@FreeBSD.org> Date: Wed, 16 Nov 2011 23:59:06 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:00:13 -0000 On 11/16/2011 23:49, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>> in a busy web hosting environment I came across the following post: >>>>>> >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>> >>>>>> That basically describes what we're seeing as well, including the >>>>>> "doesn't happen on Linux" part. >>>>>> >>>>>> Does anyone have any ideas about this? >>>>>> >>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>> so it seems to be something new in 8. >>>>> >>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>> slightly different than the one in that post. In ours the second option >>>>> is null, but the third is set: >>>>> >>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> >>>>> But repeated hundreds of times in a row. >>>> >>>> The calls cannot come from rtld, they are generated by some setjmp() >>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>> instead. >>>> >>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>> >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>> the Apache folks (not just port committers) may have some insight to >>> Kostik's findings. >> >> Thanks to everyone for the responses. We tried Kostik's suggestion and >> unfortunately it didn't reduce the number of sigprocmask() calls to a >> statistically significant degree. >> >> Does anyone have any other ideas on ways to debug this? We're sort of >> running out of things to test. :-/ >> >> Given how important (and prevalent) the Apache + FreeBSD combination is, >> I'm kind of disturbed that we're seeing this performance problem, and if >> it's something in 8.x that's also in 9.x, it would be better to fix it >> prior to 9.0-RELEASE. > > Since my guess appeared to be not useful, Well I wouldn't say that they weren't useful, we eliminated the obvious candidate. So, "not good news" certainly, but not unhelpful. :) > the way forward is to identify > the location of the call(s) that cause the issue. I suggest compliling > at least apache itself, libc, rtld and libthr (if used) with debugging > information. Then, attach to the running apache worker with the gdb and > set breakpoint on sigprocmask. Several backtraces from the hit breakpoint > should give enough data. We tried that, and got this: Loaded symbols for /libexec/ld-elf.so.1 0x28183a5d in accept () from /lib/libc.so.7 (gdb) b sigprocmask Breakpoint 1 at 0x282d8f84 (gdb) c Continuing. no thread to satisfy query 0x28183a5d in accept () from /lib/libc.so.7 (gdb) Of course I'm not the world's greatest gdb'er, so maybe there is a better way to do it? > High-tech solution is to link with libunwind and add code into sigprocmask() > to gather the stacks. But I expect that gdb attach is enough. Ok, we'll look into that, thanks. Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 08:00:24 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8EEC1065672 for ; Thu, 17 Nov 2011 08:00:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 544F98FC0C for ; Thu, 17 Nov 2011 08:00:24 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAH7nAlE009051 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Nov 2011 09:49:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAH7nAOh011871; Thu, 17 Nov 2011 09:49:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAH7n9PC011870; Thu, 17 Nov 2011 09:49:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Nov 2011 09:49:09 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="LXnoj88cJr3+5tTl" Content-Disposition: inline In-Reply-To: <4EC4ADC3.2060604@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:00:25 -0000 --LXnoj88cJr3+5tTl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > On 11/15/2011 02:09, Jeremy Chadwick wrote: > > On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > >> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > >>> On 11/14/2011 12:31, Doug Barton wrote: > >>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i= 386 > >>>> in a busy web hosting environment I came across the following post: > >>>> > >>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/23= 4520.html > >>>> > >>>> That basically describes what we're seeing as well, including the > >>>> "doesn't happen on Linux" part. > >>>> > >>>> Does anyone have any ideas about this? > >>>> > >>>> With incredibly similar stuff running on 7.x we didn't see this prob= lem, > >>>> so it seems to be something new in 8. > >>> > >>> Just took a closer look at our ktrace, and actually our pattern is > >>> slightly different than the one in that post. In ours the second opti= on > >>> is null, but the third is set: > >>> > >>> 74195 httpd 0.000017 RET sigprocmask 0 > >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> 74195 httpd 0.000009 RET sigprocmask 0 > >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> 74195 httpd 0.000009 RET sigprocmask 0 > >>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> > >>> But repeated hundreds of times in a row. > >> > >> The calls cannot come from rtld, they are generated by some setjmp() > >> invocation. If signal-safety is not needed, sigsetjmp() should be used > >> instead. > >> > >> Quick grep of the apache httpd source shows a single setjmp() in their > >> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(= ?, 0). > >=20 > > I hate cross-posting, but: adding freebsd-apache@ to the list. Some of > > the Apache folks (not just port committers) may have some insight to > > Kostik's findings. >=20 > Thanks to everyone for the responses. We tried Kostik's suggestion and > unfortunately it didn't reduce the number of sigprocmask() calls to a > statistically significant degree. >=20 > Does anyone have any other ideas on ways to debug this? We're sort of > running out of things to test. :-/ >=20 > Given how important (and prevalent) the Apache + FreeBSD combination is, > I'm kind of disturbed that we're seeing this performance problem, and if > it's something in 8.x that's also in 9.x, it would be better to fix it > prior to 9.0-RELEASE. Since my guess appeared to be not useful, the way forward is to identify the location of the call(s) that cause the issue. I suggest compliling at least apache itself, libc, rtld and libthr (if used) with debugging information. Then, attach to the running apache worker with the gdb and set breakpoint on sigprocmask. Several backtraces from the hit breakpoint should give enough data. High-tech solution is to link with libunwind and add code into sigprocmask() to gather the stacks. But I expect that gdb attach is enough. --LXnoj88cJr3+5tTl Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7EvHUACgkQC3+MBN1Mb4il/ACdE7xOw5J8y45ceLUICuABv6pc 300Ani90sb6Q2xiFuU75nGw573Ic0LSC =NOqe -----END PGP SIGNATURE----- --LXnoj88cJr3+5tTl-- From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 08:12:19 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79DE31065773; Thu, 17 Nov 2011 08:12:18 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 846A18FC17; Thu, 17 Nov 2011 08:12:18 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAH8CBlJ012106 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Nov 2011 10:12:11 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAH8CA03011993; Thu, 17 Nov 2011 10:12:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAH8CARW011992; Thu, 17 Nov 2011 10:12:10 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Nov 2011 10:12:10 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111117081210.GN50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8zu4K4iPNav6uMHi" Content-Disposition: inline In-Reply-To: <4EC4BECA.5040705@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:12:19 -0000 --8zu4K4iPNav6uMHi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: > On 11/16/2011 23:49, Kostik Belousov wrote: > > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > >> On 11/15/2011 02:09, Jeremy Chadwick wrote: > >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > >>>>> On 11/14/2011 12:31, Doug Barton wrote: > >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4= i386 > >>>>>> in a busy web hosting environment I came across the following post: > >>>>>> > >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/= 234520.html > >>>>>> > >>>>>> That basically describes what we're seeing as well, including the > >>>>>> "doesn't happen on Linux" part. > >>>>>> > >>>>>> Does anyone have any ideas about this? > >>>>>> > >>>>>> With incredibly similar stuff running on 7.x we didn't see this pr= oblem, > >>>>>> so it seems to be something new in 8. > >>>>> > >>>>> Just took a closer look at our ktrace, and actually our pattern is > >>>>> slightly different than the one in that post. In ours the second op= tion > >>>>> is null, but the third is set: > >>>>> > >>>>> 74195 httpd 0.000017 RET sigprocmask 0 > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>> > >>>>> But repeated hundreds of times in a row. > >>>> > >>>> The calls cannot come from rtld, they are generated by some setjmp() > >>>> invocation. If signal-safety is not needed, sigsetjmp() should be us= ed > >>>> instead. > >>>> > >>>> Quick grep of the apache httpd source shows a single setjmp() in the= ir > >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjm= p(?, 0). > >>> > >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some = of > >>> the Apache folks (not just port committers) may have some insight to > >>> Kostik's findings. > >> > >> Thanks to everyone for the responses. We tried Kostik's suggestion and > >> unfortunately it didn't reduce the number of sigprocmask() calls to a > >> statistically significant degree. > >> > >> Does anyone have any other ideas on ways to debug this? We're sort of > >> running out of things to test. :-/ > >> > >> Given how important (and prevalent) the Apache + FreeBSD combination i= s, > >> I'm kind of disturbed that we're seeing this performance problem, and = if > >> it's something in 8.x that's also in 9.x, it would be better to fix it > >> prior to 9.0-RELEASE. > >=20 > > Since my guess appeared to be not useful, >=20 > Well I wouldn't say that they weren't useful, we eliminated the obvious > candidate. So, "not good news" certainly, but not unhelpful. :) >=20 > > the way forward is to identify > > the location of the call(s) that cause the issue. I suggest compliling > > at least apache itself, libc, rtld and libthr (if used) with debugging > > information. Then, attach to the running apache worker with the gdb and Note this part. > > set breakpoint on sigprocmask. Several backtraces from the hit breakpoi= nt > > should give enough data. >=20 > We tried that, and got this: >=20 > Loaded symbols for /libexec/ld-elf.so.1 > 0x28183a5d in accept () from /lib/libc.so.7 > (gdb) b sigprocmask > Breakpoint 1 at 0x282d8f84 > (gdb) c > Continuing. > no thread to satisfy query > 0x28183a5d in accept () from /lib/libc.so.7 > (gdb) It seems your libc has no debugging information. accept() is the pure syscall wrapper, it cannot call sigprocmask. If gdb catched the PLT trampoline instead of real accept(), we would see the rtld frames. So install libc, libthr and rtld with debug. Also, having debug symbols for apache itself can be useful. >=20 > Of course I'm not the world's greatest gdb'er, so maybe there is a > better way to do it? >=20 > > High-tech solution is to link with libunwind and add code into sigprocm= ask() > > to gather the stacks. But I expect that gdb attach is enough. >=20 > Ok, we'll look into that, thanks. >=20 >=20 > Doug >=20 > --=20 >=20 > "We could put the whole Internet into a book." > "Too practical." >=20 > Breadth of IT experience, and depth of knowledge in the DNS. > Yours for the right price. :) http://SupersetSolutions.com/ --8zu4K4iPNav6uMHi Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7EwdoACgkQC3+MBN1Mb4jgFACgr6STI4yS2ldAPQ1t53RyC6G+ Vz8AniEJ50FpIGKMm+Kv5UgyQod4NgsM =KoyN -----END PGP SIGNATURE----- --8zu4K4iPNav6uMHi-- From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 08:34:58 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 1848F1065670; Thu, 17 Nov 2011 08:34:58 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 5C15817A7EA; Thu, 17 Nov 2011 08:34:44 +0000 (UTC) Message-ID: <4EC4C723.9090606@FreeBSD.org> Date: Thu, 17 Nov 2011 00:34:43 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Daniil Cherednik References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4C623.4090207@masterhost.ru> In-Reply-To: <4EC4C623.4090207@masterhost.ru> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Kostik Belousov , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:34:58 -0000 On 11/17/2011 00:30, Daniil Cherednik wrote: > I am sorry for repeat (I wrote about it), but what do you think about > this hack: Danill, thanks, and sorry if I wasn't clear before, but the problem we're seeing has a very clear pattern: 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) That the rtld calls don't exhibit. Kostik, thanks for your more detailed response, we'll poke that a bit and report back. Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 09:27:15 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id F17081065670; Thu, 17 Nov 2011 09:27:15 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 110C4175F70; Thu, 17 Nov 2011 09:26:49 +0000 (UTC) Message-ID: <4EC4D359.4040406@FreeBSD.org> Date: Thu, 17 Nov 2011 01:26:49 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117081210.GN50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 09:27:16 -0000 On 11/17/2011 00:12, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: >> On 11/16/2011 23:49, Kostik Belousov wrote: >>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>>>> in a busy web hosting environment I came across the following post: >>>>>>>> >>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>>>> >>>>>>>> That basically describes what we're seeing as well, including the >>>>>>>> "doesn't happen on Linux" part. >>>>>>>> >>>>>>>> Does anyone have any ideas about this? >>>>>>>> >>>>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>>>> so it seems to be something new in 8. >>>>>>> >>>>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>>>> slightly different than the one in that post. In ours the second option >>>>>>> is null, but the third is set: >>>>>>> >>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> >>>>>>> But repeated hundreds of times in a row. >>>>>> >>>>>> The calls cannot come from rtld, they are generated by some setjmp() >>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>>>> instead. >>>>>> >>>>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>>>> >>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>>>> the Apache folks (not just port committers) may have some insight to >>>>> Kostik's findings. >>>> >>>> Thanks to everyone for the responses. We tried Kostik's suggestion and >>>> unfortunately it didn't reduce the number of sigprocmask() calls to a >>>> statistically significant degree. >>>> >>>> Does anyone have any other ideas on ways to debug this? We're sort of >>>> running out of things to test. :-/ >>>> >>>> Given how important (and prevalent) the Apache + FreeBSD combination is, >>>> I'm kind of disturbed that we're seeing this performance problem, and if >>>> it's something in 8.x that's also in 9.x, it would be better to fix it >>>> prior to 9.0-RELEASE. >>> >>> Since my guess appeared to be not useful, >> >> Well I wouldn't say that they weren't useful, we eliminated the obvious >> candidate. So, "not good news" certainly, but not unhelpful. :) >> >>> the way forward is to identify >>> the location of the call(s) that cause the issue. I suggest compliling >>> at least apache itself, libc, rtld and libthr (if used) with debugging >>> information. Then, attach to the running apache worker with the gdb and > Note this part. Right, we attached to a worker, that's why it's in accept(). :) > It seems your libc has no debugging information. > accept() is the pure syscall wrapper, it cannot call sigprocmask. > If gdb catched the PLT trampoline instead of real accept(), we would > see the rtld frames. So install libc, libthr and rtld with debug. It's not catching there though: Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) info threads Cannot get thread info: invalid key (gdb) Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 10:19:01 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A8EF1065670 for ; Thu, 17 Nov 2011 10:19:01 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta12.westchester.pa.mail.comcast.net (qmta12.westchester.pa.mail.comcast.net [76.96.59.227]) by mx1.freebsd.org (Postfix) with ESMTP id E70BE8FC14 for ; Thu, 17 Nov 2011 10:19:00 +0000 (UTC) Received: from omta09.westchester.pa.mail.comcast.net ([76.96.62.20]) by qmta12.westchester.pa.mail.comcast.net with comcast id yAG31h0030SCNGk5CAK1vN; Thu, 17 Nov 2011 10:19:01 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta09.westchester.pa.mail.comcast.net with comcast id yAJy1h00E1t3BNj3VAJymM; Thu, 17 Nov 2011 10:19:00 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id DAC05102C1D; Thu, 17 Nov 2011 02:18:56 -0800 (PST) Date: Thu, 17 Nov 2011 02:18:56 -0800 From: Jeremy Chadwick To: Kostik Belousov Message-ID: <20111117101856.GA39096@icarus.home.lan> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111117081210.GN50300@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, Daniil Cherednik , Doug Barton , freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:19:01 -0000 On Thu, Nov 17, 2011 at 10:12:10AM +0200, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: > > On 11/16/2011 23:49, Kostik Belousov wrote: > > > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > > >> On 11/15/2011 02:09, Jeremy Chadwick wrote: > > >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > > >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > > >>>>> On 11/14/2011 12:31, Doug Barton wrote: > > >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 > > >>>>>> in a busy web hosting environment I came across the following post: > > >>>>>> > > >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html > > >>>>>> > > >>>>>> That basically describes what we're seeing as well, including the > > >>>>>> "doesn't happen on Linux" part. > > >>>>>> > > >>>>>> Does anyone have any ideas about this? > > >>>>>> > > >>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, > > >>>>>> so it seems to be something new in 8. > > >>>>> > > >>>>> Just took a closer look at our ktrace, and actually our pattern is > > >>>>> slightly different than the one in that post. In ours the second option > > >>>>> is null, but the third is set: > > >>>>> > > >>>>> 74195 httpd 0.000017 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> > > >>>>> But repeated hundreds of times in a row. > > >>>> > > >>>> The calls cannot come from rtld, they are generated by some setjmp() > > >>>> invocation. If signal-safety is not needed, sigsetjmp() should be used > > >>>> instead. > > >>>> > > >>>> Quick grep of the apache httpd source shows a single setjmp() in their > > >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). > > >>> > > >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of > > >>> the Apache folks (not just port committers) may have some insight to > > >>> Kostik's findings. > > >> > > >> Thanks to everyone for the responses. We tried Kostik's suggestion and > > >> unfortunately it didn't reduce the number of sigprocmask() calls to a > > >> statistically significant degree. > > >> > > >> Does anyone have any other ideas on ways to debug this? We're sort of > > >> running out of things to test. :-/ > > >> > > >> Given how important (and prevalent) the Apache + FreeBSD combination is, > > >> I'm kind of disturbed that we're seeing this performance problem, and if > > >> it's something in 8.x that's also in 9.x, it would be better to fix it > > >> prior to 9.0-RELEASE. > > > > > > Since my guess appeared to be not useful, > > > > Well I wouldn't say that they weren't useful, we eliminated the obvious > > candidate. So, "not good news" certainly, but not unhelpful. :) > > > > > the way forward is to identify > > > the location of the call(s) that cause the issue. I suggest compliling > > > at least apache itself, libc, rtld and libthr (if used) with debugging > > > information. Then, attach to the running apache worker with the gdb and > Note this part. > > > > set breakpoint on sigprocmask. Several backtraces from the hit breakpoint > > > should give enough data. > > > > We tried that, and got this: > > > > Loaded symbols for /libexec/ld-elf.so.1 > > 0x28183a5d in accept () from /lib/libc.so.7 > > (gdb) b sigprocmask > > Breakpoint 1 at 0x282d8f84 > > (gdb) c > > Continuing. > > no thread to satisfy query > > 0x28183a5d in accept () from /lib/libc.so.7 > > (gdb) > It seems your libc has no debugging information. > accept() is the pure syscall wrapper, it cannot call sigprocmask. > If gdb catched the PLT trampoline instead of real accept(), we would > see the rtld frames. So install libc, libthr and rtld with debug. > > Also, having debug symbols for apache itself can be useful. I'd also like to point out that enabling debugging symbols in devel/apr1 will be greatly needed here, not just in www/apache*. I'm wondering if maybe this is some sort of pthread "thing" going on. A quick grep -r sigmask of the Apache source turns up some pthread_* bits pertaining to worker. Is Apache build using WITH_THREADS? What about devel/apr1? I don't use worker MPM on any of our boxes, we actually use ITK MPM solely because of the hosting nature of what we do. I've actually never seen worker MPM in use on any *IX machine I've been on or administrated, only prefork. The Apache documentation even mentions that "if you want stability or compatibility, prefork is the choice", while "if you want scalability, worker is a better choice"[1]. These sorts of quotes often shock me given what year it is. :-) [1]: http://httpd.apache.org/docs/2.0/mpm.html -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 10:24:08 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D83F2106564A; Thu, 17 Nov 2011 10:24:08 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 750018FC08; Thu, 17 Nov 2011 10:24:08 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:d9a7:15b3:2d22:93e6]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 659E54AC2D; Thu, 17 Nov 2011 14:24:06 +0400 (MSK) Date: Thu, 17 Nov 2011 14:24:01 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <11310157468.20111117142401@serebryakov.spb.ru> To: Kostik Belousov In-Reply-To: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org, Daniil Cherednik , Doug Barton , freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:24:08 -0000 Hello, Kostik. You wrote 17 =ED=EE=FF=E1=F0=FF 2011 =E3., 11:49:09: > High-tech solution is to link with libunwind and add code into sigprocmas= k() > to gather the stacks. But I expect that gdb attach is enough. Proper high-tech solution is to use DTrace. It is very food in such things. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 10:57:49 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2387106564A; Thu, 17 Nov 2011 10:57:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 674268FC16; Thu, 17 Nov 2011 10:57:48 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAHAvib6042499 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAHAviYf012457; Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAHAviCC012456; Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Nov 2011 12:57:44 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111117105744.GS50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pSPXDt+5DZRK1gNs" Content-Disposition: inline In-Reply-To: <4EC4D359.4040406@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:57:49 -0000 --pSPXDt+5DZRK1gNs Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Nov 17, 2011 at 01:26:49AM -0800, Doug Barton wrote: > On 11/17/2011 00:12, Kostik Belousov wrote: > > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: > >> On 11/16/2011 23:49, Kostik Belousov wrote: > >>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > >>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: > >>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > >>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > >>>>>>> On 11/14/2011 12:31, Doug Barton wrote: > >>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-= p4 i386 > >>>>>>>> in a busy web hosting environment I came across the following po= st: > >>>>>>>> > >>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-Octobe= r/234520.html > >>>>>>>> > >>>>>>>> That basically describes what we're seeing as well, including the > >>>>>>>> "doesn't happen on Linux" part. > >>>>>>>> > >>>>>>>> Does anyone have any ideas about this? > >>>>>>>> > >>>>>>>> With incredibly similar stuff running on 7.x we didn't see this = problem, > >>>>>>>> so it seems to be something new in 8. > >>>>>>> > >>>>>>> Just took a closer look at our ktrace, and actually our pattern is > >>>>>>> slightly different than the one in that post. In ours the second = option > >>>>>>> is null, but the third is set: > >>>>>>> > >>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> > >>>>>>> But repeated hundreds of times in a row. > >>>>>> > >>>>>> The calls cannot come from rtld, they are generated by some setjmp= () > >>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be = used > >>>>>> instead. > >>>>>> > >>>>>> Quick grep of the apache httpd source shows a single setjmp() in t= heir > >>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigset= jmp(?, 0). > >>>>> > >>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Som= e of > >>>>> the Apache folks (not just port committers) may have some insight to > >>>>> Kostik's findings. > >>>> > >>>> Thanks to everyone for the responses. We tried Kostik's suggestion a= nd > >>>> unfortunately it didn't reduce the number of sigprocmask() calls to a > >>>> statistically significant degree. > >>>> > >>>> Does anyone have any other ideas on ways to debug this? We're sort of > >>>> running out of things to test. :-/ > >>>> > >>>> Given how important (and prevalent) the Apache + FreeBSD combination= is, > >>>> I'm kind of disturbed that we're seeing this performance problem, an= d if > >>>> it's something in 8.x that's also in 9.x, it would be better to fix = it > >>>> prior to 9.0-RELEASE. > >>> > >>> Since my guess appeared to be not useful, > >> > >> Well I wouldn't say that they weren't useful, we eliminated the obvious > >> candidate. So, "not good news" certainly, but not unhelpful. :) > >> > >>> the way forward is to identify > >>> the location of the call(s) that cause the issue. I suggest compliling > >>> at least apache itself, libc, rtld and libthr (if used) with debugging > >>> information. Then, attach to the running apache worker with the gdb a= nd > > Note this part. >=20 > Right, we attached to a worker, that's why it's in accept(). :) >=20 > > It seems your libc has no debugging information. > > accept() is the pure syscall wrapper, it cannot call sigprocmask. > > If gdb catched the PLT trampoline instead of real accept(), we would > > see the rtld frames. So install libc, libthr and rtld with debug. >=20 > It's not catching there though: >=20 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > 0x28183b2d in accept () at accept.S:3 > 3 RSYSCALL(accept) > (gdb) c > Continuing. > no thread to satisfy query > 0x28183b2d in accept () at accept.S:3 > 3 RSYSCALL(accept) > (gdb) info threads > Cannot get thread info: invalid key > (gdb) Err, the other part of my message was that you shall set the breakpoint on sigprocmask. I want to see a backtrace from the breakpoint hit. Several times. The backtrace at the attach time has no use. --pSPXDt+5DZRK1gNs Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7E6KgACgkQC3+MBN1Mb4j5mgCgvbV20mLT2co6NO3NUTQlM8Ub kOQAmwU4tRvdIjYTtMfkfVwUq63h/pLe =pZru -----END PGP SIGNATURE----- --pSPXDt+5DZRK1gNs-- From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 11:32:18 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E74CF106566C; Thu, 17 Nov 2011 11:32:18 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6D1BE8FC12; Thu, 17 Nov 2011 11:32:18 +0000 (UTC) Received: by vcbfy13 with SMTP id fy13so1402489vcb.13 for ; Thu, 17 Nov 2011 03:32:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=aUWxZO5AMLb7NVvMyZyAuq0p5HCuPfZ5CWO+DNcuUo8=; b=ImBBA3+sjrCdK5ZdsHk2Y3aG5osHKfQ8odxX4viIX9WFOgA8CSL4ERw0K30kuP/Ro7 knSPUYUNBXNuB6po7i97QbCqtJPqTusLaietlRxyFsMhOS9ecNzvc/liU19AOD8cC4fC MqxERxg9WAy5jr9kVymMx1ZlR4SUXVj74n9lQ= MIME-Version: 1.0 Received: by 10.52.38.6 with SMTP id c6mr57265309vdk.73.1321527951777; Thu, 17 Nov 2011 03:05:51 -0800 (PST) Received: by 10.52.182.40 with HTTP; Thu, 17 Nov 2011 03:05:51 -0800 (PST) In-Reply-To: <20111117101856.GA39096@icarus.home.lan> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <20111117101856.GA39096@icarus.home.lan> Date: Thu, 17 Nov 2011 11:05:51 +0000 Message-ID: From: Tom Evans To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Kostik Belousov , Daniil Cherednik , Doug Barton , freebsd-stable@freebsd.org, freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 11:32:19 -0000 On Thu, Nov 17, 2011 at 10:18 AM, Jeremy Chadwick wrote: > I don't use worker MPM on any of our boxes, we actually use ITK MPM > solely because of the hosting nature of what we do. =C2=A0I've actually n= ever > seen worker MPM in use on any *IX machine I've been on or administrated, > only prefork. =C2=A0The Apache documentation even mentions that "if you w= ant > stability or compatibility, prefork is the choice", while "if you want > scalability, worker is a better choice"[1]. =C2=A0These sorts of quotes o= ften > shock me given what year it is. =C2=A0:-) > I've used both worker and event MPMs in production on high volume sites for > 4 years now, running on FreeBSD 7, with no problems. I think you are cherry picking the quotes from httpd's 2.0 documentation, which is actually an old bit of software now - it has just been voted EOL. The current stable (2.2) docs actually say: "sites that need a great deal of scalability can choose to use a threaded MPM like worker or event, while sites requiring stability or compatibility with older software can use a prefork" Event and worker have no issues unless you run non thread safe modules, or modules which use libraries which are not thread safe, eg PHP (more commonly, a PHP extension). Cheers Tom From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 08:41:30 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5F41106564A for ; Thu, 17 Nov 2011 08:41:30 +0000 (UTC) (envelope-from dcherednik@masterhost.ru) Received: from mail.corp.masterhost.ru (wincas02.mail.corp.masterhost.ru [90.156.219.212]) by mx1.freebsd.org (Postfix) with ESMTP id DA4DA8FC13 for ; Thu, 17 Nov 2011 08:41:29 +0000 (UTC) Received: from kzholnay-pc.masterhost.ru (87.242.97.5) by mail.corp.masterhost.ru (90.156.219.210) with Microsoft SMTP Server (TLS) id 14.1.339.1; Thu, 17 Nov 2011 12:30:28 +0400 Message-ID: <4EC4C623.4090207@masterhost.ru> Date: Thu, 17 Nov 2011 12:30:27 +0400 From: Daniil Cherednik User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 MIME-Version: 1.0 To: References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [87.242.97.5] X-Mailman-Approved-At: Thu, 17 Nov 2011 11:54:11 +0000 Cc: Kostik Belousov , Doug Barton , freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:41:30 -0000 On 17.11.2011 11:49, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>> in a busy web hosting environment I came across the following post: >>>>>> >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>> >>>>>> That basically describes what we're seeing as well, including the >>>>>> "doesn't happen on Linux" part. >>>>>> >>>>>> Does anyone have any ideas about this? >>>>>> >>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>> so it seems to be something new in 8. >>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>> slightly different than the one in that post. In ours the second option >>>>> is null, but the third is set: >>>>> >>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> >>>>> But repeated hundreds of times in a row. >>>> The calls cannot come from rtld, they are generated by some setjmp() >>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>> instead. >>>> >>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>> the Apache folks (not just port committers) may have some insight to >>> Kostik's findings. >> Thanks to everyone for the responses. We tried Kostik's suggestion and >> unfortunately it didn't reduce the number of sigprocmask() calls to a >> statistically significant degree. >> >> Does anyone have any other ideas on ways to debug this? We're sort of >> running out of things to test. :-/ >> >> Given how important (and prevalent) the Apache + FreeBSD combination is, >> I'm kind of disturbed that we're seeing this performance problem, and if >> it's something in 8.x that's also in 9.x, it would be better to fix it >> prior to 9.0-RELEASE. > Since my guess appeared to be not useful, the way forward is to identify > the location of the call(s) that cause the issue. I suggest compliling > at least apache itself, libc, rtld and libthr (if used) with debugging > information. Then, attach to the running apache worker with the gdb and > set breakpoint on sigprocmask. Several backtraces from the hit breakpoint > should give enough data. > > High-tech solution is to link with libunwind and add code into sigprocmask() > to gather the stacks. But I expect that gdb attach is enough. I am sorry for repeat (I wrote about it), but what do you think about this hack: diff -u ./rtld_lock.c.orig ./rtld_lock.c --- ./rtld_lock.c.orig 2011-11-15 07:56:14.000000000 +0000 +++ ./rtld_lock.c 2011-11-15 07:54:42.000000000 +0000 @@ -118,7 +118,7 @@ sigset_t tmp_oldsigmask; for ( ; ; ) { - sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); +// sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); if (atomic_cmpset_acq_int(&l->lock, 0, WAFLAG)) break; sigprocmask(SIG_SETMASK, &tmp_oldsigmask, NULL); @@ -135,7 +135,7 @@ atomic_add_rel_int(&l->lock, -RC_INCR); else { atomic_add_rel_int(&l->lock, -WAFLAG); - sigprocmask(SIG_SETMASK, &oldsigmask, NULL); +// sigprocmask(SIG_SETMASK, &oldsigmask, NULL); } } This is one of source sigprocmask. Look: truss with original /libexec/ld-elf.so.1: #truss true __sysctl(0xbfbfe624,0x2,0xbfbfe62c,0xbfbfe630,0x0,0x0) = 0 (0x0) mmap(0x0,320,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 671633408 (0x28085000) munmap(0x28085000,320) = 0 (0x0) __sysctl(0xbfbfe688,0x2,0x2807be3c,0xbfbfe690,0x0,0x0) = 0 (0x0) mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 671633408 (0x28085000) issetugid(0x28074967,0xbfbfeb4c,0x104,0x0,0x0,0x0) = 0 (0x0) open("/etc/libmap.conf",O_RDONLY,0666) ERR#2 'No such file or directory' open("/var/run/ld-elf.so.hints",O_RDONLY,00) = 3 (0x3) read(3,"Ehnt\^A\0\0\0\M^@\0\0\0\M^E\0\0"...,128) = 128 (0x80) lseek(3,0x80,SEEK_SET) = 128 (0x80) read(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,133) = 133 (0x85) close(3) = 0 (0x0) access("/lib/libc.so.7",0) = 0 (0x0) open("/lib/libc.so.7",O_RDONLY,00) = 3 (0x3) fstat(3,{ mode=-r--r--r-- ,inode=94234,size=1155172,blksize=16384 }) = 0 (0x0) pread(0x3,0x2807ad80,0x1000,0x0,0x0,0x0) = 4096 (0x1000) mmap(0x0,1159168,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) = 671666176 (0x2808d000) mmap(0x2808d000,1040384,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE,3,0x0) = 671666176 (0x2808d000) mmap(0x2818b000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xfe000) = 672706560 (0x2818b000) mprotect(0x28191000,94208,PROT_READ|PROT_WRITE) = 0 (0x0) close(3) = 0 (0x0) sysarch(0xa,0xbfbfe6f0,0x2804b3cb,0x2807a2f8,0x2805de39,0x2807a2f8) = 0 (0x0) mmap(0x0,64,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 672825344 (0x281a8000) munmap(0x281a8000,64) = 0 (0x0) mmap(0x0,21896,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 672825344 (0x281a8000) munmap(0x281a8000,21896) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) __sysctl(0xbfbfe6a4,0x2,0x28192700,0xbfbfe6ac,0x0,0x0) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) process exit, rval = 0 and truss after patching: #truss true __sysctl(0xbfbfe5a4,0x2,0xbfbfe5ac,0xbfbfe5b0,0x0,0x0) = 0 (0x0) mmap(0x0,320,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 671633408 (0x28085000) munmap(0x28085000,320) = 0 (0x0) __sysctl(0xbfbfe608,0x2,0x2807be3c,0xbfbfe610,0x0,0x0) = 0 (0x0) mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 671633408 (0x28085000) issetugid(0x28074927,0xbfbfeacc,0x104,0x0,0x0,0x0) = 0 (0x0) open("/etc/libmap.conf",O_RDONLY,0666) ERR#2 'No such file or directory' open("/var/run/ld-elf.so.hints",O_RDONLY,00) = 3 (0x3) read(3,"Ehnt\^A\0\0\0\M^@\0\0\0B\0\0\0\0"...,128) = 128 (0x80) lseek(3,0x80,SEEK_SET) = 128 (0x80) read(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,66) = 66 (0x42) close(3) = 0 (0x0) access("/lib/libc.so.7",0) = 0 (0x0) open("/lib/libc.so.7",O_RDONLY,00) = 3 (0x3) fstat(3,{ mode=-r--r--r-- ,inode=12,size=1155172,blksize=16384 }) = 0 (0x0) pread(0x3,0x2807ad80,0x1000,0x0,0x0,0x0) = 4096 (0x1000) mmap(0x0,1159168,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) = 671666176 (0x2808d000) mmap(0x2808d000,1040384,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE,3,0x0) = 671666176 (0x2808d000) mmap(0x2818b000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xfe000) = 672706560 (0x2818b000) mprotect(0x28191000,94208,PROT_READ|PROT_WRITE) = 0 (0x0) close(3) = 0 (0x0) sysarch(0xa,0xbfbfe670,0x2804b3cb,0x2807a2f8,0x2805ddf9,0x2807a2f8) = 0 (0x0) mmap(0x0,64,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 672825344 (0x281a8000) munmap(0x281a8000,64) = 0 (0x0) mmap(0x0,21896,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 672825344 (0x281a8000) munmap(0x281a8000,21896) = 0 (0x0) __sysctl(0xbfbfe624,0x2,0x28192700,0xbfbfe62c,0x0,0x0) = 0 (0x0) process exit, rval = 0 I am not expert in the internal structure of ld-elf.so, but I am trying to think there is more effective way for locks. Because now we are doing a few syscalls for each dynamic loaded library in fork time. -- С уважением, Daniil Cherednik .masterhost From owner-freebsd-apache@FreeBSD.ORG Thu Nov 17 12:39:43 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 50225106564A; Thu, 17 Nov 2011 12:39:43 +0000 (UTC) (envelope-from dcherednik@masterhost.ru) Received: from mail.corp.masterhost.ru (wincas02.mail.corp.masterhost.ru [90.156.219.212]) by mx1.freebsd.org (Postfix) with ESMTP id 78C198FC15; Thu, 17 Nov 2011 12:39:42 +0000 (UTC) Received: from kzholnay-pc.masterhost.ru (87.242.97.5) by mail.corp.masterhost.ru (90.156.219.210) with Microsoft SMTP Server (TLS) id 14.1.339.1; Thu, 17 Nov 2011 16:39:40 +0400 Message-ID: <4EC5008B.8050603@masterhost.ru> Date: Thu, 17 Nov 2011 16:39:39 +0400 From: Daniil Cherednik User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 MIME-Version: 1.0 To: References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <20111117101856.GA39096@icarus.home.lan> In-Reply-To: <20111117101856.GA39096@icarus.home.lan> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [87.242.97.5] X-Mailman-Approved-At: Thu, 17 Nov 2011 12:47:47 +0000 Cc: Kostik Belousov , Doug Barton , freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 12:39:43 -0000 On 17.11.2011 14:18, Jeremy Chadwick wrote: > On Thu, Nov 17, 2011 at 10:12:10AM +0200, Kostik Belousov wrote: >> On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: >>> On 11/16/2011 23:49, Kostik Belousov wrote: >>>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >>>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>>>>> in a busy web hosting environment I came across the following post: >>>>>>>>> >>>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>>>>> >>>>>>>>> That basically describes what we're seeing as well, including the >>>>>>>>> "doesn't happen on Linux" part. >>>>>>>>> >>>>>>>>> Does anyone have any ideas about this? >>>>>>>>> >>>>>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>>>>> so it seems to be something new in 8. >>>>>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>>>>> slightly different than the one in that post. In ours the second option >>>>>>>> is null, but the third is set: >>>>>>>> >>>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> >>>>>>>> But repeated hundreds of times in a row. >>>>>>> The calls cannot come from rtld, they are generated by some setjmp() >>>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>>>>> instead. >>>>>>> >>>>>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>>>>> the Apache folks (not just port committers) may have some insight to >>>>>> Kostik's findings. >>>>> Thanks to everyone for the responses. We tried Kostik's suggestion and >>>>> unfortunately it didn't reduce the number of sigprocmask() calls to a >>>>> statistically significant degree. >>>>> >>>>> Does anyone have any other ideas on ways to debug this? We're sort of >>>>> running out of things to test. :-/ >>>>> >>>>> Given how important (and prevalent) the Apache + FreeBSD combination is, >>>>> I'm kind of disturbed that we're seeing this performance problem, and if >>>>> it's something in 8.x that's also in 9.x, it would be better to fix it >>>>> prior to 9.0-RELEASE. >>>> Since my guess appeared to be not useful, >>> Well I wouldn't say that they weren't useful, we eliminated the obvious >>> candidate. So, "not good news" certainly, but not unhelpful. :) >>> >>>> the way forward is to identify >>>> the location of the call(s) that cause the issue. I suggest compliling >>>> at least apache itself, libc, rtld and libthr (if used) with debugging >>>> information. Then, attach to the running apache worker with the gdb and >> Note this part. >> >>>> set breakpoint on sigprocmask. Several backtraces from the hit breakpoint >>>> should give enough data. >>> We tried that, and got this: >>> >>> Loaded symbols for /libexec/ld-elf.so.1 >>> 0x28183a5d in accept () from /lib/libc.so.7 >>> (gdb) b sigprocmask >>> Breakpoint 1 at 0x282d8f84 >>> (gdb) c >>> Continuing. >>> no thread to satisfy query >>> 0x28183a5d in accept () from /lib/libc.so.7 >>> (gdb) >> It seems your libc has no debugging information. >> accept() is the pure syscall wrapper, it cannot call sigprocmask. >> If gdb catched the PLT trampoline instead of real accept(), we would >> see the rtld frames. So install libc, libthr and rtld with debug. >> >> Also, having debug symbols for apache itself can be useful. > I'd also like to point out that enabling debugging symbols in devel/apr1 > will be greatly needed here, not just in www/apache*. > > I'm wondering if maybe this is some sort of pthread "thing" going on. A > quick grep -r sigmask of the Apache source turns up some pthread_* bits > pertaining to worker. > > Is Apache build using WITH_THREADS? What about devel/apr1? > > I don't use worker MPM on any of our boxes, we actually use ITK MPM > solely because of the hosting nature of what we do. I've actually never > seen worker MPM in use on any *IX machine I've been on or administrated, > only prefork. The Apache documentation even mentions that "if you want > stability or compatibility, prefork is the choice", while "if you want > scalability, worker is a better choice"[1]. These sorts of quotes often > shock me given what year it is. :-) > > [1]: http://httpd.apache.org/docs/2.0/mpm.html > We use ITK MPM too, but we have big trouble with performance on FreeBSD. Also, I have to say we can`t use keep-alive connection, so apache creates new child for each request. -- С уважением, Daniil Cherednik .masterhost From owner-freebsd-apache@FreeBSD.ORG Fri Nov 18 08:01:01 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 792E6106564A; Fri, 18 Nov 2011 08:01:01 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 8C54114E685; Fri, 18 Nov 2011 08:00:57 +0000 (UTC) Message-ID: <4EC610B9.8080406@FreeBSD.org> Date: Fri, 18 Nov 2011 00:00:57 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> <20111117105744.GS50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117105744.GS50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 08:01:01 -0000 On 11/17/2011 02:57, Kostik Belousov wrote: >> > It's not catching there though: >> > >> > Reading symbols from /libexec/ld-elf.so.1...done. >> > Loaded symbols for /libexec/ld-elf.so.1 >> > 0x28183b2d in accept () at accept.S:3 >> > 3 RSYSCALL(accept) >> > (gdb) c >> > Continuing. >> > no thread to satisfy query >> > 0x28183b2d in accept () at accept.S:3 >> > 3 RSYSCALL(accept) >> > (gdb) info threads >> > Cannot get thread info: invalid key >> > (gdb) > Err, the other part of my message was that you shall set the breakpoint > on sigprocmask. I'm sorry I'm not making myself clear. We are setting the breakpoint on sigprocmask. But, maybe I'm doing it wrong. Can you give precise instructions as to what you want me to do, from the beginning? Sorry to be so dense. > I want to see a backtrace from the breakpoint hit. > Several times. Me too. :) Meanwhile, in response to one of the other questions, we are using mpm_prefork. Also, the particular problem we're seeing does not appear related to fork(). The pattern of sigprocmask() calls is different from the pattern you see with fork(). Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-apache@FreeBSD.ORG Fri Nov 18 09:19:42 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 151BF106566B; Fri, 18 Nov 2011 09:19:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id A21F48FC0C; Fri, 18 Nov 2011 09:19:41 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAI9Jboh001798 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 18 Nov 2011 11:19:37 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAI9Jb8P016837; Fri, 18 Nov 2011 11:19:37 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAI9JbX3016836; Fri, 18 Nov 2011 11:19:37 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 18 Nov 2011 11:19:37 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111118091937.GA50300@deviant.kiev.zoral.com.ua> References: <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> <20111117105744.GS50300@deviant.kiev.zoral.com.ua> <4EC610B9.8080406@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZEJo5p0yAM1jsdkh" Content-Disposition: inline In-Reply-To: <4EC610B9.8080406@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 09:19:42 -0000 --ZEJo5p0yAM1jsdkh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 18, 2011 at 12:00:57AM -0800, Doug Barton wrote: > On 11/17/2011 02:57, Kostik Belousov wrote: > >> > It's not catching there though: > >> >=20 > >> > Reading symbols from /libexec/ld-elf.so.1...done. > >> > Loaded symbols for /libexec/ld-elf.so.1 > >> > 0x28183b2d in accept () at accept.S:3 > >> > 3 RSYSCALL(accept) > >> > (gdb) c > >> > Continuing. > >> > no thread to satisfy query > >> > 0x28183b2d in accept () at accept.S:3 > >> > 3 RSYSCALL(accept) > >> > (gdb) info threads > >> > Cannot get thread info: invalid key > >> > (gdb) > > Err, the other part of my message was that you shall set the breakpoint > > on sigprocmask. >=20 > I'm sorry I'm not making myself clear. We are setting the breakpoint on > sigprocmask. But, maybe I'm doing it wrong. Can you give precise > instructions as to what you want me to do, from the beginning? Sorry to > be so dense. Find the pid of the process issuing excessive number of sigprocmask calls. Do $ gdb /usr/local/bin/httpd (gdb) attach (gdb) b _sigprocmask (gdb) c Bah ! Breakpoint fired. (gdb) bt (gdb) c <... Repeat ... > >=20 > > I want to see a backtrace from the breakpoint hit. > > Several times. >=20 > Me too. :) >=20 > Meanwhile, in response to one of the other questions, we are using > mpm_prefork. Also, the particular problem we're seeing does not appear > related to fork(). The pattern of sigprocmask() calls is different from > the pattern you see with fork(). I am sure that your sigprocmask calls do not come from rtld, it is some use of setjmp or sigsetjmp(1), most likely. I am not aware of any significant users of setjmp or sigprocmask in our system libraries. >=20 >=20 > Doug >=20 > --=20 >=20 > "We could put the whole Internet into a book." > "Too practical." >=20 > Breadth of IT experience, and depth of knowledge in the DNS. > Yours for the right price. :) http://SupersetSolutions.com/ --ZEJo5p0yAM1jsdkh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7GIykACgkQC3+MBN1Mb4gPtwCdEwGLoFiLVJsgTwpmRYNkeXd0 yuEAnj5wdYlX5mJ9r/3oWnVWsQviyC5J =tAQC -----END PGP SIGNATURE----- --ZEJo5p0yAM1jsdkh-- From owner-freebsd-apache@FreeBSD.ORG Fri Nov 18 20:07:51 2011 Return-Path: Delivered-To: freebsd-apache@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id DCB351065673; Fri, 18 Nov 2011 20:07:51 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 5E157160720; Fri, 18 Nov 2011 20:07:51 +0000 (UTC) Message-ID: <4EC6BB17.20706@FreeBSD.org> Date: Fri, 18 Nov 2011 12:07:51 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> <20111117105744.GS50300@deviant.kiev.zoral.com.ua> <4EC610B9.8080406@FreeBSD.org> <20111118091937.GA50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111118091937.GA50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-apache@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Support of apache-related ports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 20:07:52 -0000 On 11/18/2011 01:19, Kostik Belousov wrote: > On Fri, Nov 18, 2011 at 12:00:57AM -0800, Doug Barton wrote: >> On 11/17/2011 02:57, Kostik Belousov wrote: >>>>> It's not catching there though: >>>>> >>>>> Reading symbols from /libexec/ld-elf.so.1...done. >>>>> Loaded symbols for /libexec/ld-elf.so.1 >>>>> 0x28183b2d in accept () at accept.S:3 >>>>> 3 RSYSCALL(accept) >>>>> (gdb) c >>>>> Continuing. >>>>> no thread to satisfy query >>>>> 0x28183b2d in accept () at accept.S:3 >>>>> 3 RSYSCALL(accept) >>>>> (gdb) info threads >>>>> Cannot get thread info: invalid key >>>>> (gdb) >>> Err, the other part of my message was that you shall set the breakpoint >>> on sigprocmask. >> >> I'm sorry I'm not making myself clear. We are setting the breakpoint on >> sigprocmask. But, maybe I'm doing it wrong. Can you give precise >> instructions as to what you want me to do, from the beginning? Sorry to >> be so dense. > Find the pid of the process issuing excessive number of sigprocmask > calls. Do > $ gdb /usr/local/bin/httpd > (gdb) attach > (gdb) b _sigprocmask > (gdb) c > Bah ! Breakpoint fired. > (gdb) bt > (gdb) c > <... Repeat ... > Right, so we're on the same page at least. I've been abbreviating the output of gdb to make it easier to see the problem, but here is a (nearly) complete transcript: gdb /usr/local/bin/httpd Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... (gdb) attach 1380 Attaching to program: /usr/local/bin/httpd, process 1380 Reading symbols from .... (lots of symbol-reading snipped) 3 RSYSCALL(accept) Current language: auto; currently asm (gdb) b _sigprocmask Breakpoint 1 at 0x282d9055: file /usr/src/lib/libthr/thread/thr_sig.c, line 210. (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) .... etc. -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/