From owner-freebsd-python@FreeBSD.ORG Mon Jun 9 11:07:05 2008 Return-Path: Delivered-To: freebsd-python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 12F531065674 for ; Mon, 9 Jun 2008 11:07:05 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id ED8B98FC1E for ; Mon, 9 Jun 2008 11:07:04 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m59B74Fc070852 for ; Mon, 9 Jun 2008 11:07:04 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m59B743G070848 for freebsd-python@FreeBSD.org; Mon, 9 Jun 2008 11:07:04 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 9 Jun 2008 11:07:04 GMT Message-Id: <200806091107.m59B743G070848@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-python@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-python@FreeBSD.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2008 11:07:05 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- a ports/108576 python databases/postgresql-plpython make fails on 6.2 / amd6 o ports/116132 python lang/Python25 - Python 2.4 -> 2.5 upgrade leaves meta- 2 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o ports/109550 python lang/python25 does not install /usr/local/bin/python o ports/110728 python mail/py-spambayes should load config file from a stand o ports/115940 python Missed one file in lang/python25 if NO_NIS defined o ports/118301 python devel/py-setuptools easy-install.pth contents lost on o ports/119440 python lang/python25: NO_NIS=true in /etc/make.conf breaks pk o ports/123154 python New port: devel/py-yapps2 (an easy-to-use parser gener f ports/123627 python [PATCH]devel/py-distorm: update to 1.7.29 and rework p o ports/123675 python New port: deskutils/py-labyrinth Mind mapping tool for f ports/123797 python Update port: devel/py-tables to 2.0.3 9 problems total. From owner-freebsd-python@FreeBSD.ORG Mon Jun 9 11:08:17 2008 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEC981065681 for ; Mon, 9 Jun 2008 11:08:17 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8CB6C8FC27 for ; Mon, 9 Jun 2008 11:08:17 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m59B8HMx072292 for ; Mon, 9 Jun 2008 11:08:17 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m59B8Hvp072288 for python@FreeBSD.org; Mon, 9 Jun 2008 11:08:17 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 9 Jun 2008 11:08:17 GMT Message-Id: <200806091108.m59B8Hvp072288@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: python@FreeBSD.org Cc: Subject: Current problem reports assigned to python@FreeBSD.org X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2008 11:08:17 -0000 Current FreeBSD problem reports Critical problems Serious problems Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o ports/122616 python databases/py-pyPgSQL - apply bytea escape bug patch in 1 problem total. From owner-freebsd-python@FreeBSD.ORG Tue Jun 10 16:51:25 2008 Return-Path: Delivered-To: python@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4944D106566B; Tue, 10 Jun 2008 16:51:25 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 18E078FC1E; Tue, 10 Jun 2008 16:51:23 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <484EB10D.8090003@FreeBSD.org> Date: Tue, 10 Jun 2008 18:51:25 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: cperciva@FreeBSD.org, glewis@FreeBSD.org, hq@FreeBSD.org, python@FreeBSD.org Content-Type: multipart/mixed; boundary="------------090605040602090901090603" Cc: Subject: [Fwd: INDEX build optimizations - please review] X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2008 16:51:25 -0000 This is a multi-part message in MIME format. --------------090605040602090901090603 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Hi guys, Since you are all positive "stakeholders" in the code I'm changing, I'd like you to sign off on it before I proceed. Thanks, Kris --------------090605040602090901090603 Content-Type: message/rfc822; name="INDEX build optimizations - please review.eml" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="INDEX build optimizations - please review.eml" X-Account-Key: account2 X-Mozilla-Keys: Return-Path: X-Original-To: kris@freebsd.org Delivered-To: kris@freebsd.org Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 1609C10656CE; Tue, 10 Jun 2008 16:45:38 +0000 (UTC) (envelope-from owner-freebsd-ports@freebsd.org) Delivered-To: ports@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37554106564A for ; Tue, 10 Jun 2008 16:45:34 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BA8FC8FC14 for ; Tue, 10 Jun 2008 16:45:31 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <484EAFAC.3020208@FreeBSD.org> Date: Tue, 10 Jun 2008 18:45:32 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: ports@FreeBSD.org Content-Type: multipart/mixed; boundary="------------040303030706050704010102" Cc: Subject: INDEX build optimizations - please review X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: owner-freebsd-ports@freebsd.org Errors-To: owner-freebsd-ports@freebsd.org This is a multi-part message in MIME format. --------------040303030706050704010102 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Please review and test the following patches that optimize port INDEX builds (and as a side-effect, other recursive tree traversals). I am particularly interested in a comparison between old and new indexes built locally: the only diff should be in audio/festvox-hvs [1]. The patches remove most of the extraneous command executions required for each invocation of bsd.port.mk (e.g. via 'make describe' in a port), by replacing external command invocations with shell or make builtins, as well as caching of constants. Another important optimization is to use /rescue/sh instead of /bin/sh for index builds. The former is statically linked and this is much faster to execute. One further change I didn't do would be to move the WWW specification from pkg-descr into a Makefile variable (this could be done mechanically). The new 'make describe' target runs entirely using shell builtins apart from the need to sed pkg-descr to extract the WWW [2] (previously 'describe' was a combination of several shell executions and a perl script, invoked for every port). With these patches an index build on an 8 core system drops from 796.486u 974.564s 5:25.14 544.7% 28+193k 37252+719io 27pf+0w to 642.846u 164.520s 2:31.29 533.6% 67+297k 0+721io 0pf+0w (or with statically linked sed: 637.805u 142.335s 2:27.32 529.5% 71+304k 0+720io 0pf+0w ) The new version requires 69301 forks and 68614 execs (would be ~19000 fewer of each without WWW in pkg-descr), compared to 252383 forks and 226875 execs with the current version. The resulting index is identical except for one port [1] but generating it is more than twice as fast (and uses 6 times less system CPU). I am only getting ~530% CPU utilization because of contention in the scheduler, so there is scope for going as low as 100 seconds if this went away. Further improvements are no doubt possible but would require profiling the work done within make(1) to see where it is spending its time (variable setting, conditional evaluation, loop invocation, regexp processing, etc). ********************************************************************** ********************* NOTE TO PORT DEVELOPERS ************************ ********************************************************************** Variable assignments with != are bad! Try as hard as you can to avoid using them -- especially in Mk/*! Every time something processes your makefile it will spawn a command, even if it is not relevant for the operation being performed. If you need to run shell commands, try to isolate them within a makefile target. You can avoid code duplication by assigning the *shell commands* (not their output) to a variable and inserting it into your code block. e.g. instead of -- VARIABLE!= do some shell stuff; do some other stuff target: echo ${VARIABLE} -- do this (or similar): -- VARIABLE_CMDS= do some shell stuff; do some other stuff target: echo $(${VARIABLE_CMDS}) -- This defers the command execution to the point where the target runs, so in the case when the target is *not* run, then you avoid wasting one or more process executions. Kris [1] This patch exposed a bug: for some reason no dependencies were previously recorded for audio/festvox-hvs! Probably because it only has RUN_DEPENDS and the 'make describe' perl script was broken. [2] Actually I am not happy with this but couldn't think of a way to do it better. Having to fork the subshell costs about 60 seconds of system time and 10 of wall time. --------------040303030706050704010102 Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0"; name="index-fast.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="index-fast.diff" Index: Makefile =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Makefile,v retrieving revision 1.103 diff -u -r1.103 Makefile --- Makefile 27 Sep 2007 05:36:26 -0000 1.103 +++ Makefile 10 Jun 2008 09:56:17 -0000 @@ -103,6 +103,7 @@ tmpdir=`/usr/bin/mktemp -d -t index` || exit 1; \ trap "rm -rf $${tmpdir}; exit 1" 1 2 3 5 10 13 15; \ ( cd ${.CURDIR} && make -j${INDEX_JOBS} INDEX_TMPDIR=$${tmpdir} BUILDING_INDEX=1 \ + __MAKE_SHELL=/rescue/sh \ ECHO_MSG="${INDEX_ECHO_MSG}" describe ) || \ (rm -rf $${tmpdir} ; \ if [ "${INDEX_QUIET}" = "" ]; then \ Index: Mk/bsd.java.mk =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.java.mk,v retrieving revision 1.82 diff -u -r1.82 bsd.java.mk --- Mk/bsd.java.mk 28 Oct 2007 15:09:43 -0000 1.82 +++ Mk/bsd.java.mk 10 Jun 2008 09:56:17 -0000 @@ -249,34 +249,30 @@ . endfor # Error checking: JAVA_VERSION +.if !defined(_JAVA_VERSION_LIST_REGEXP) _JAVA_VERSION_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_VERSION_LIST}" | ${SED} "s/ /\\\|/g" -_ERROR_CHECKING_JAVA_VERSION!= ${ECHO_CMD} "${JAVA_VERSION}" | ${TR} " " "\n" \ - | ${GREP} -v "${_JAVA_VERSION_LIST_REGEXP}" || true -. if (${_ERROR_CHECKING_JAVA_VERSION} != "") +.endif check-makevars:: - @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VERSION}\" is not a valid value for JAVA_VERSION. It should be one or more of: ${__JAVA_VERSION_LIST} (with an optional \"+\" suffix.)"; - @${FALSE} -. endif + @${ECHO_CMD} "${JAVA_VERSION}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_VERSION_LIST_REGEXP}" || \ + (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VERSION}\" is not a valid value for JAVA_VERSION. It should be one or more of: ${__JAVA_VERSION_LIST} (with an optional \"+\" suffix.)"; ${FALSE}) # Error checking: JAVA_VENDOR +.if !defined(_JAVA_VENDOR_LIST_REGEXP) _JAVA_VENDOR_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_VENDOR_LIST}" | ${SED} "s/ /\\\|/g" -_ERROR_CHECKING_JAVA_VENDOR!= ${ECHO_CMD} "${JAVA_VENDOR}" | ${TR} " " "\n" \ - | ${GREP} -v "${_JAVA_VENDOR_LIST_REGEXP}" || true -. if (${_ERROR_CHECKING_JAVA_VENDOR} != "") +.endif check-makevars:: - @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VENDOR}\" is not a valid value for JAVA_VENDOR. It should be one or more of: ${_JAVA_VENDOR_LIST}"; - @${FALSE} -. endif + @${ECHO_CMD} "${JAVA_VENDOR}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_VENDOR_LIST_REGEXP}" || \ + (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VENDOR}\" is not a valid value for JAVA_VENDOR. It should be one or more of: ${_JAVA_VENDOR_LIST}"; \ + ${FALSE}) # Error checking: JAVA_OS +.if !defined(_JAVA_OS_LIST_REGEXP) _JAVA_OS_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_OS_LIST}" | ${SED} "s/ /\\\|/g" -_ERROR_CHECKING_JAVA_OS!= ${ECHO_CMD} "${JAVA_OS}" | ${TR} " " "\n" \ - | ${GREP} -v "${_JAVA_OS_LIST_REGEXP}" || true -. if (${_ERROR_CHECKING_JAVA_OS} != "") +.endif check-makevars:: - @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_OS}\" is not a valid value for JAVA_OS. It should be one or more of: ${_JAVA_OS_LIST}"; - @${FALSE} -. endif + @${ECHO_CMD} "${JAVA_OS}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_OS_LIST_REGEXP}" || \ + (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_OS}\" is not a valid value for JAVA_OS. It should be one or more of: ${_JAVA_OS_LIST}"; \ + ${FALSE}) # Set default values for JAVA_BUILD and JAVA_RUN # When nothing is set, assume JAVA_BUILD=jdk and JAVA_RUN=jre @@ -311,20 +307,32 @@ A_JAVA_PORT_INFO:= ${A_JAVA_PORT:S/^/\${_/:S/$/_INFO}/} A_JAVA_PORT_HOME= ${A_JAVA_PORT_INFO:MHOME=*:S,HOME=,,} A_JAVA_PORT_VERSION= ${A_JAVA_PORT_INFO:MVERSION=*:C/VERSION=([0-9])\.([0-9])(.*)/\1.\2/} -A_JAVA_PORT_OS= ${A_JAVA_PORT_INFO:MOS=*:S,OS=,,} -A_JAVA_PORT_VENDOR= ${A_JAVA_PORT_INFO:MVENDOR=*:S,VENDOR=,,} +A_JAVA_PORT_OS= ${A_JAVA_PORT_INFO:MOS=*:S,OS=,,} +A_JAVA_PORT_VENDOR= ${A_JAVA_PORT_INFO:MVENDOR=*:S,VENDOR=,,} +.if !defined(_JAVA_PORTS_INSTALLED) A_JAVA_PORT_INSTALLED!= ${TEST} -x "${A_JAVA_PORT_HOME}/${_JDK_FILE}" \ && ${ECHO_CMD} "${A_JAVA_PORT}" \ || ${TRUE} __JAVA_PORTS_INSTALLED!= ${ECHO_CMD} "${__JAVA_PORTS_INSTALLED} ${A_JAVA_PORT_INSTALLED}" -A_JAVA_PORT_POSSIBLE!= ${ECHO_CMD} "${_JAVA_VERSION}" | ${GREP} -q "${A_JAVA_PORT_VERSION}" \ - && ${ECHO_CMD} "${_JAVA_OS}" | ${GREP} -q "${A_JAVA_PORT_OS}" \ - && ${ECHO_CMD} "${_JAVA_VENDOR}" | ${GREP} -q "${A_JAVA_PORT_VENDOR}" \ - && ${ECHO_CMD} "${A_JAVA_PORT}" \ - || ${TRUE} -__JAVA_PORTS_POSSIBLE!= ${ECHO_CMD} "${__JAVA_PORTS_POSSIBLE} ${A_JAVA_PORT_POSSIBLE}" +.endif + +# The magic here is that we want to test for a substring using only shell builtins (to avoid forking) +# Our shell does not have an explicit substring operator, but we can build one by using the '#' +# deletion operator ('%' would also work). We try to delete the pattern "*${substr}*" and compare it +# to the original string. If they differ, the substring matched. +# +# We can't do this in make because it doesn't allow nested modifiers ${foo:${bar}} +# +A_JAVA_PORT_POSSIBLE!= ver="${_JAVA_VERSION}"; os="${_JAVA_OS}"; vendor="${_JAVA_VENDOR}"; \ + ${TEST} "$${ver\#*${A_JAVA_PORT_VERSION}*}" != "${_JAVA_VERSION}" -a \ + "$${os\#*${A_JAVA_PORT_OS}*}" != "${_JAVA_OS}" -a \ + "$${vendor\#*${A_JAVA_PORT_VENDOR}*}" != "${_JAVA_VENDOR}" && \ + ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE} +__JAVA_PORTS_POSSIBLE:= ${__JAVA_PORTS_POSSIBLE} ${A_JAVA_PORT_POSSIBLE} . endfor +.if !defined(_JAVA_PORTS_INSTALLED) _JAVA_PORTS_INSTALLED= ${__JAVA_PORTS_INSTALLED:C/ [ ]+/ /g} +.endif _JAVA_PORTS_POSSIBLE= ${__JAVA_PORTS_POSSIBLE:C/ [ ]+/ /g} @@ -337,20 +345,28 @@ . undef _JAVA_PORTS_INSTALLED_POSSIBLE . for A_JAVA_PORT in ${_JAVA_PORTS_POSSIBLE} -A_JAVA_PORT_INSTALLED_POSSIBLE!= ${ECHO_CMD} "${_JAVA_PORTS_INSTALLED}" | ${GREP} -q "${A_JAVA_PORT}" \ - && ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE} -__JAVA_PORTS_INSTALLED_POSSIBLE!= ${ECHO_CMD} "${__JAVA_PORTS_INSTALLED_POSSIBLE} ${A_JAVA_PORT_INSTALLED_POSSIBLE}" +A_JAVA_PORT_INSTALLED_POSSIBLE!= inst="${_JAVA_PORTS_INSTALLED}"; \ + ${TEST} "$${inst\#*${A_JAVA_PORT}*}" != "${_JAVA_PORTS_INSTALLED}" && \ + ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE} +__JAVA_PORTS_INSTALLED_POSSIBLE:= ${__JAVA_PORTS_INSTALLED_POSSIBLE} ${A_JAVA_PORT_INSTALLED_POSSIBLE} . endfor -_JAVA_PORTS_INSTALLED_POSSIBLE= ${__JAVA_PORTS_INSTALLED_POSSIBLE:C/ [ ]+/ /g} +_JAVA_PORTS_INSTALLED_POSSIBLE= ${__JAVA_PORTS_INSTALLED_POSSIBLE:C/[ ]+//g} . if ${_JAVA_PORTS_INSTALLED_POSSIBLE} != "" -_JAVA_PORT!= ${ECHO_CMD} "${_JAVA_PORTS_INSTALLED_POSSIBLE}" \ - | ${AWK} '{ print $$1 }' - +. for i in ${_JAVA_PORTS_INSTALLED_POSSIBLE} +. if !defined(_JAVA_PORTS_INSTALLED_POSSIBLE_shortcircuit) +_JAVA_PORT= $i +_JAVA_PORTS_INSTALLED_POSSIBLE_shortcircuit= 1 +. endif +. endfor # If no installed JDK port fits, then pick one from the list of possible ones . else -_JAVA_PORT!= ${ECHO_CMD} "${_JAVA_PORTS_POSSIBLE}" \ - | ${AWK} '{ print $$1 }' +. for i in ${_JAVA_PORTS_POSSIBLE} +. if !defined(_JAVA_PORTS_POSSIBLE_shortcircuit) +_JAVA_PORT= $i +_JAVA_PORTS_POSSIBLE_shortcircuit= 1 +. endif +. endfor . endif _JAVA_PORT_INFO:= ${_JAVA_PORT:S/^/\${_/:S/$/_INFO}/} Index: Mk/bsd.port.mk =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.port.mk,v retrieving revision 1.593 diff -u -r1.593 bsd.port.mk --- Mk/bsd.port.mk 27 May 2008 22:12:02 -0000 1.593 +++ Mk/bsd.port.mk 10 Jun 2008 11:08:58 -0000 @@ -1235,14 +1239,13 @@ UNIQUENAME?= ${PKGNAMEPREFIX}${PORTNAME} .endif OPTIONSFILE?= ${PORT_DBDIR}/${UNIQUENAME}/options -_OPTIONSFILE!= ${ECHO_CMD} "${OPTIONSFILE}" .if defined(OPTIONS) # include OPTIONSFILE first if exists -. if exists(${_OPTIONSFILE}) && !make(rmconfig) -. include "${_OPTIONSFILE}" +. if exists(${OPTIONSFILE}) && !make(rmconfig) +. include "${OPTIONSFILE}" . endif -. if exists(${_OPTIONSFILE}.local) -. include "${_OPTIONSFILE}.local" +. if exists(${OPTIONSFILE}.local) +. include "${OPTIONSFILE}.local" . endif WITHOUT:= WITH:= @@ -1640,7 +1643,7 @@ HAVE_COMPAT_IA32_LIBS?= YES .endif .if !defined(HAVE_COMPAT_IA32_KERN) -HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -a compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi +HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -n compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi .endif .endif @@ -3265,8 +3268,8 @@ .endif .if defined(_OPTIONS_READ) @${ECHO_MSG} "===> Found saved configuration for ${_OPTIONS_READ}" -.if ${OPTIONSFILE} != ${_OPTIONSFILE} - @${ECHO_MSG} "===> *** CAUTION *** Using wrong configuration file ${_OPTIONSFILE}" +.if ${OPTIONSFILE} != ${OPTIONSFILE} + @${ECHO_MSG} "===> *** CAUTION *** Using wrong configuration file ${OPTIONSFILE}" .endif .endif @@ -5283,68 +5286,29 @@ # first to avoid gratuitous breakage. .if !target(describe) +_EXTRACT_DEPENDS=${EXTRACT_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} +_PATCH_DEPENDS=${PATCH_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} +_FETCH_DEPENDS=${FETCH_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} +_LIB_DEPENDS=${LIB_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} +_BUILD_DEPENDS=${BUILD_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} ${_LIB_DEPENDS} +_RUN_DEPENDS=${RUN_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} ${_LIB_DEPENDS} +.if exists(${DESCR}) +_DESCR=${DESCR} +.else +_DESCR=/dev/null +.endif + describe: - @${ECHO_CMD} -n "${PKGNAME}|${.CURDIR}|${PREFIX}|" -.if defined(COMMENT) - @${ECHO_CMD} -n ${COMMENT:Q} -.else - @${ECHO_CMD} -n '** No Description' -.endif - @perl -e ' \ - if ( -f q{${DESCR}} ) { \ - print q{|${DESCR}}; \ - } else { \ - print q{|/dev/null}; \ - } \ - print q{|${MAINTAINER}|${CATEGORIES}|}; \ - @edirs = map((split /:/)[1], split(q{ }, q{${EXTRACT_DEPENDS}})); \ - @pdirs = map((split /:/)[1], split(q{ }, q{${PATCH_DEPENDS}})); \ - @fdirs = map((split /:/)[1], split(q{ }, q{${FETCH_DEPENDS}})); \ - @bdirs = map((split /:/)[1], split(q{ }, q{${BUILD_DEPENDS}})); \ - @rdirs = map((split /:/)[1], split(q{ }, q{${RUN_DEPENDS}})); \ - @ldirs = map((split /:/)[1], split(q{ }, q{${LIB_DEPENDS}})); \ - for my $$i (\@edirs, \@pdirs, \@fdirs, \@bdirs, \@rdirs, \@ddirs, \@ldirs) { \ - my @dirs = @$$i; \ - @$$i = (); \ - for (@dirs) { \ - if (-d $$_) { \ - push @$$i, $$_; \ - } else { \ - print STDERR qq{${PKGNAME}: \"$$_\" non-existent -- dependency list incomplete\n}; \ - exit(1); \ - } \ - } \ - } \ - for (@edirs, @ddirs) { \ - $$xe{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xe), q{|}; \ - for (@pdirs, @ddirs) { \ - $$xp{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xp), q{|}; \ - for (@fdirs, @ddirs) { \ - $$xf{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xf), q{|}; \ - for (@bdirs, @ddirs, @ldirs) { \ - $$xb{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xb), q{|}; \ - for (@rdirs, @ddirs, @ldirs) { \ - $$xr{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xr), q{|}; \ - if (open(DESCR, q{${DESCR}})) { \ - while () { \ - if (/^WWW:\s+(\S+)/) { \ - print $$1; \ - last; \ - } \ - } \ - } \ - print qq{\n};' + @${ECHO_CMD} -n "${PKGNAME}|${.CURDIR}|${PREFIX}|"; \ + ${ECHO_CMD} -n ${COMMENT:Q}; \ + ${ECHO_CMD} -n "|${_DESCR}|${MAINTAINER}|${CATEGORIES}|${_EXTRACT_DEPENDS}|${_PATCH_DEPENDS}|${_FETCH_DEPENDS}|${_BUILD_DEPENDS:O:u}|${_RUN_DEPENDS:O:u}|"; \ + set "" $$(sed -E -e '/^WWW:[[:blank:]]+/!d' -e 's,^WWW:[[:blank:]]+([[:print:]]+).*$$,\1,' ${_DESCR}); \ + echo $$2 .endif +# | (read site && ${ECHO_CMD} $${site}) || ${ECHO_CMD} +# sed -E -e '/^WWW:/!d' -e 's,^WWW:.*(http[:print:]*),\1,' ${_DESCR} | (read site && ${ECHO_CMD} $${site}) + +# ${GREP} -m1 "^WWW:" ${_DESCR} | (read foo site bar && ${ECHO_CMD} $$site) || ${ECHO_CMD} www-site: .if exists(${DESCR}) @@ -5768,23 +5732,23 @@ .if !defined(OPTIONS) @${ECHO_MSG} "===> No options to configure" .else -.if ${OPTIONSFILE} != ${_OPTIONSFILE} - @${ECHO_MSG} "===> Using wrong configuration file ${_OPTIONSFILE}" +.if ${OPTIONSFILE} != ${OPTIONSFILE} + @${ECHO_MSG} "===> Using wrong configuration file ${OPTIONSFILE}" @exit 1 .endif .if ${UID} != 0 && !defined(INSTALL_AS_USER) - @optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ + @optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ ${ECHO_MSG} "===> Switching to root credentials to create $${optionsdir}"; \ (${SU_CMD} "${SH} -c \"${MKDIR} $${optionsdir} 2> /dev/null\"") || \ (${ECHO_MSG} "===> Cannot create $${optionsdir}, check permissions"; exit 1); \ ${ECHO_MSG} "===> Returning to user credentials" .else - @(optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ + @(optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ ${MKDIR} $${optionsdir} 2> /dev/null) || \ (${ECHO_MSG} "===> Cannot create $${optionsdir}, check permissions"; exit 1) .endif - -@if [ -e ${_OPTIONSFILE} ]; then \ - . ${_OPTIONSFILE}; \ + -@if [ -e ${OPTIONSFILE} ]; then \ + . ${OPTIONSFILE}; \ fi; \ set -- ${OPTIONS} XXX; \ while [ $$# -gt 3 ]; do \ @@ -5834,11 +5798,11 @@ fi; \ done; \ if [ `${ID} -u` != 0 -a "x${INSTALL_AS_USER}" = "x" ]; then \ - ${ECHO_MSG} "===> Switching to root credentials to write ${_OPTIONSFILE}"; \ - ${SU_CMD} "${CAT} $${TMPOPTIONSFILE} > ${_OPTIONSFILE}"; \ + ${ECHO_MSG} "===> Switching to root credentials to write ${OPTIONSFILE}"; \ + ${SU_CMD} "${CAT} $${TMPOPTIONSFILE} > ${OPTIONSFILE}"; \ ${ECHO_MSG} "===> Returning to user credentials"; \ else \ - ${CAT} $${TMPOPTIONSFILE} > ${_OPTIONSFILE}; \ + ${CAT} $${TMPOPTIONSFILE} > ${OPTIONSFILE}; \ fi; \ ${RM} -f $${TMPOPTIONSFILE} .endif @@ -5855,9 +5819,9 @@ .if !target(config-conditional) config-conditional: .if defined(OPTIONS) -.if exists(${_OPTIONSFILE}) +.if exists(${OPTIONSFILE}) # scan saved options and invalidate them, if the set of options does not match - @. ${_OPTIONSFILE}; \ + @. ${OPTIONSFILE}; \ set ${OPTIONS} XXX; \ while [ $$# -gt 3 ]; do \ withvar=WITH_$$1; \ @@ -5889,8 +5853,8 @@ showconfig: .if defined(OPTIONS) @${ECHO_MSG} "===> The following configuration options are available for ${PKGNAME}:" - -@if [ -e ${_OPTIONSFILE} ]; then \ - . ${_OPTIONSFILE}; \ + -@if [ -e ${OPTIONSFILE} ]; then \ + . ${OPTIONSFILE}; \ fi; \ set -- ${OPTIONS} XXX; \ while [ $$# -gt 3 ]; do \ @@ -5915,16 +5879,16 @@ .if !target(rmconfig) rmconfig: -.if defined(OPTIONS) && exists(${_OPTIONSFILE}) +.if defined(OPTIONS) && exists(${OPTIONSFILE}) -@${ECHO_MSG} "===> Removing user-configured options for ${PKGNAME}"; \ - optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ + optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ if [ `${ID} -u` != 0 -a "x${INSTALL_AS_USER}" = "x" ]; then \ - ${ECHO_MSG} "===> Switching to root credentials to remove ${_OPTIONSFILE} and $${optionsdir}"; \ - ${SU_CMD} "${RM} -f ${_OPTIONSFILE} ; \ + ${ECHO_MSG} "===> Switching to root credentials to remove ${OPTIONSFILE} and $${optionsdir}"; \ + ${SU_CMD} "${RM} -f ${OPTIONSFILE} ; \ ${RMDIR} $${optionsdir}"; \ ${ECHO_MSG} "===> Returning to user credentials"; \ else \ - ${RM} -f ${_OPTIONSFILE}; \ + ${RM} -f ${OPTIONSFILE}; \ ${RMDIR} $${optionsdir}; \ fi .else Index: Mk/bsd.port.subdir.mk =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.port.subdir.mk,v retrieving revision 1.74 diff -u -r1.74 bsd.port.subdir.mk --- Mk/bsd.port.subdir.mk 12 Mar 2008 00:13:06 -0000 1.74 +++ Mk/bsd.port.subdir.mk 10 Jun 2008 11:59:30 -0000 @@ -63,13 +63,15 @@ STRIP?= -s .endif +# These are variables that are invariant for the lifetime of a recursive port traversal +# (index build, etc), so it is more efficient to precompute them here and pass them in +# to child makes explicitly, instead of recomputing them tens of thousands of times. + .if !defined(NOPRECIOUSMAKEVARS) .if !defined(ARCH) ARCH!= ${UNAME} -p .endif -.if !defined(OSREL) -OSREL!= ${UNAME} -r | ${SED} -e 's/[-(].*//' -.endif + .if !defined(OSVERSION) .if exists(/usr/include/sys/param.h) OSVERSION!= ${AWK} '/^\#define[[:blank:]]__FreeBSD_version/ {print $$3}' < /usr/include/sys/param.h @@ -79,23 +81,76 @@ OSVERSION!= ${SYSCTL} -n kern.osreldate .endif .endif + +.if !defined(_OSRELEASE) +_OSRELEASE!= uname -r +.endif +.if !defined(OSREL) +OSREL= ${_OSRELEASE:C/[-(].*//} .endif -INDEXDIR?= ${PORTSDIR} -INDEXFILE?= INDEX-${OSVERSION:C/([0-9]).*/\1/} +.if !defined(OPSYS) +OPSYS!= ${UNAME} -s +.endif + +.if ${ARCH} == "amd64" || ${ARCH} =="ia64" +.if !defined(HAVE_COMPAT_IA32_KERN) +HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -n compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi +.endif +.endif +.if !defined(CONFIGURE_MAX_CMD_LEN) +CONFIGURE_MAX_CMD_LEN!= ${SYSCTL} -n kern.argmax +.endif + +.if !defined(PYTHON_DEFAULT_VERSION) +PYTHON_DEFAULT_VERSION!= make -V PYTHON_DEFAULT_VERSION USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(PYTHON_DEFAULT_PORTVERSION) +# We are caching the PYTHON_PORTVERSION of the default python version so we can reuse it in the +# common case. +PYTHON_DEFAULT_PORTVERSION!= make -V PYTHON_PORTVERSION USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(PYTHONBASE) +PYTHONBASE!= make -V PYTHONBASE USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(_JAVA_VERSION_LIST_REGEXP) +_JAVA_VERSION_LIST_REGEXP!= make -V _JAVA_VERSION_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(_JAVA_VENDOR_LIST_REGEXP) +_JAVA_VENDOR_LIST_REGEXP!= make -V _JAVA_VENDOR_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(_JAVA_OS_LIST_REGEXP) +_JAVA_OS_LIST_REGEXP!= make -V _JAVA_OS_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(_JAVA_PORTS_INSTALLED) +_JAVA_PORTS_INSTALLED!= make -V _JAVA_PORTS_INSTALLED USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(UID) UID!= ${ID} -u +.endif + .if exists(${LOCALBASE}/sbin/pkg_info) PKG_INFO?= ${LOCALBASE}/sbin/pkg_info .else PKG_INFO?= /usr/sbin/pkg_info .endif +.if !defined(PKGINSTALLVER) PKGINSTALLVER!= ${PKG_INFO} -P 2>/dev/null | ${SED} -e 's/.*: //' +.endif -.if !defined(OPSYS) -OPSYS!= ${UNAME} -s .endif +INDEXDIR?= ${PORTSDIR} +INDEXFILE?= INDEX-${OSVERSION:C/([0-9]).*/\1/} + # local customization of the ports tree .if exists(${.CURDIR}/Makefile.local) .include "${.CURDIR}/Makefile.local" @@ -307,6 +362,8 @@ > $@ @${RM} -f $@.tmp $@.tmp2 $@.tmp3 $@.tmp4 +# Pass in the cached invariant variables to child makes. +# XXX Why are we trying to escape these characters using regexps and not using ':Q'? .if !defined(NOPRECIOUSMAKEVARS) .MAKEFLAGS: \ ARCH="${ARCH:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ @@ -314,7 +371,16 @@ OSREL="${OSREL:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ OSVERSION="${OSVERSION:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ UID="${UID:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ - PKGINSTALLVER="${PKGINSTALLVER:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" + PKGINSTALLVER="${PKGINSTALLVER:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ + HAVE_COMPAT_IA32_KERN="${HAVE_COMPAT_IA32_KERN}" \ + CONFIGURE_MAX_CMD_LEN="${CONFIGURE_MAX_CMD_LEN}" \ + PYTHON_DEFAULT_VERSION="${PYTHON_DEFAULT_VERSION}" \ + PYTHON_DEFAULT_PORTVERSION="${PYTHON_DEFAULT_PORTVERSION}" \ + PYTHONBASE="${PYTHONBASE}" \ + _JAVA_VERSION_LIST_REGEXP="${_JAVA_VERSION_LIST_REGEXP}" \ + _JAVA_VENDOR_LIST_REGEXP="${_JAVA_VENDOR_LIST_REGEXP}" \ + _JAVA_OS_LIST_REGEXP="${_JAVA_OS_LIST_REGEXP}" \ + _JAVA_PORTS_INSTALLED="${_JAVA_PORTS_INSTALLED}" .endif PORTSEARCH_DISPLAY_FIELDS?=name,path,info,maint,index,bdeps,rdeps,www Index: Mk/bsd.python.mk =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.python.mk,v retrieving revision 1.104 diff -u -r1.104 bsd.python.mk --- Mk/bsd.python.mk 30 May 2008 05:50:03 -0000 1.104 +++ Mk/bsd.python.mk 10 Jun 2008 12:08:49 -0000 @@ -352,14 +352,24 @@ PYTHON_VERSION?= python${_PYTHON_VERSION} PYTHON_CMD?= ${_PYTHON_CMD} +.if !defined(PYTHONBASE) PYTHONBASE!= (${PYTHON_CMD} -c 'import sys; print sys.prefix' \ 2> /dev/null || ${ECHO_CMD} ${LOCALBASE}) | ${TAIL} -1 +.endif DEPENDS_ARGS+= PYTHON_VERSION=${PYTHON_VERSION} + +# We can only use the cached version if we are using the default python version. Otherwise it +# should point to some other version we have installed, according to the port USE_PYTHON +# specification +.if !defined(PYTHON_DEFAULT_PORTVERSION) || (${PYTHON_VERSION} != ${PYTHON_DEFAULT_VERSION}) _PYTHON_PORTVERSION!= (${PYTHON_CMD} -c 'import string, sys; \ print string.split(sys.version)[0].replace("b",".b")' 2> /dev/null) | ${TAIL} -1 .if !defined(PYTHON_NO_DEPENDS) && !empty(_PYTHON_PORTVERSION) PYTHON_PORTVERSION= ${_PYTHON_PORTVERSION} .endif +.elif defined(PYTHON_DEFAULT_PORTVERSION) +PYTHON_PORTVERSION= ${PYTHON_DEFAULT_PORTVERSION} +.endif # Propagate the chosen python version to submakes. .MAKEFLAGS: PYTHON_VERSION=python${_PYTHON_VERSION} @@ -440,7 +450,9 @@ .endif .if defined(PYEASYINSTALL_ARCHDEP) +.if !defined(_OSRELEASE) _OSRELEASE!= ${UNAME} -r +.endif PYEASYINSTALL_OSARCH?= -${OPSYS:L}-${_OSRELEASE}-${ARCH} .endif PYEASYINSTALL_EGG?= ${PYDISTUTILS_PKGNAME:C/[^A-Za-z0-9.]+/_/g}-${PYDISTUTILS_PKGVERSION:C/[^A-Za-z0-9.]+/_/g}-${PYTHON_VERSION:S/thon//}${PYEASYINSTALL_OSARCH}.egg --------------040303030706050704010102 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" --------------040303030706050704010102-- --------------090605040602090901090603-- From owner-freebsd-python@FreeBSD.ORG Thu Jun 12 01:06:33 2008 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 939B6106564A for ; Thu, 12 Jun 2008 01:06:33 +0000 (UTC) (envelope-from felipebgn@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.168]) by mx1.freebsd.org (Postfix) with ESMTP id 1D98D8FC0C for ; Thu, 12 Jun 2008 01:06:32 +0000 (UTC) (envelope-from felipebgn@gmail.com) Received: by ug-out-1314.google.com with SMTP id q2so254382uge.37 for ; Wed, 11 Jun 2008 18:06:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=aKH1Rig6dQDz4f1++PEt86kRj6EwSu8u0DZ5TcTMHc0=; b=nhla02c5QL/T+onGHcPGe99VfFoqNB+jfAPhICddA3uV25oK9WJUWtQ43IQh2IFYTE lhrsl7A8axIuKIhiOV1p2vY3/nW/X2RzuZHzM/gCcshva/igS8RRGDT3G0u5dKTONdxZ ZV477aYe3y2BWsylT+ujb+NBjWK3/odazJ9G8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=I74IVKzf349lI+OJag7u60QJSBN5+MMH1xvsNIDqi3O62voTGEAGdAh8KWV9tG0Hvi iGM5ORarim0kihG86EYef5LGTkkCFPxQLxGGn2r0r4KPHrGO39cgIHDsRbH2Oo+q8fNj 2CnbhkNIvlMto1a6WucfD+cEJ549SEhdB5OrA= Received: by 10.210.19.11 with SMTP id 11mr436168ebs.111.1213231110244; Wed, 11 Jun 2008 17:38:30 -0700 (PDT) Received: by 10.210.19.16 with HTTP; Wed, 11 Jun 2008 17:38:30 -0700 (PDT) Message-ID: <928b5da90806111738h55bbbbb0y3e9731323a1561f4@mail.gmail.com> Date: Wed, 11 Jun 2008 21:38:30 -0300 From: "Felipe Neuwald" To: freebsd-performance@freebsd.org, freebsd-python@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Cc: Subject: Performance with python and FreeBSD 7.0 amd64 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2008 01:06:33 -0000 Hi all, We have a few servers running zope + plone. On one server running FreeBSD 6.3-STABLE i386, I got no problems, but, with one server running FreeBSD 7.0-STABLE amd64, same versions of applications, I got some errors, like the following: dmesg result: pid 74775 (python), uid 1002: exited on signal 11 pid 74861 (python), uid 1002: exited on signal 11 pid 74911 (python), uid 1002: exited on signal 11 pid 74926 (python), uid 1002: exited on signal 11 pid 74970 (python), uid 1002: exited on signal 11 pid 75038 (python), uid 1002: exited on signal 11 pid 75069 (python), uid 1002: exited on signal 11 pid 75095 (python), uid 1002: exited on signal 11 pid 75131 (python), uid 1002: exited on signal 11 pid 75136 (python), uid 1002: exited on signal 11 pid 75204 (python), uid 1002: exited on signal 11 pid 75842 (python), uid 1002: exited on signal 11 pid 75949 (python), uid 1002: exited on signal 10 pid 75962 (python), uid 1002: exited on signal 11 pid 75999 (python), uid 1002: exited on signal 4 pid 76097 (python), uid 1002: exited on signal 10 pid 77452 (python), uid 1002: exited on signal 11 pid 78012 (python), uid 1002: exited on signal 10 pid 78044 (python), uid 1002: exited on signal 11 pid 78425 (python), uid 1002: exited on signal 4 pid 78464 (python), uid 1002: exited on signal 11 pid 78615 (python), uid 1002: exited on signal 11 pid 78638 (python), uid 1002: exited on signal 11 pid 78656 (python), uid 1002: exited on signal 11 pid 78809 (python), uid 1002: exited on signal 11 pid 79076 (python), uid 1002: exited on signal 11 pid 84577 (python), uid 1002: exited on signal 11 I'm using python version 2.4 on every server. So, if someone can help me: - How can I debug the system to find the error? - How can I configure the server for plone + zope (python) best performance? Thank you very much, Felipe Neuwald. From owner-freebsd-python@FreeBSD.ORG Thu Jun 12 12:40:43 2008 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6A01B106568F; Thu, 12 Jun 2008 12:40:43 +0000 (UTC) (envelope-from andymac@bullseye.apana.org.au) Received: from ipmail01.adl6.internode.on.net (ipmail01.adl6.internode.on.net [203.16.214.146]) by mx1.freebsd.org (Postfix) with ESMTP id C4A4C8FC1D; Thu, 12 Jun 2008 12:40:42 +0000 (UTC) (envelope-from andymac@bullseye.apana.org.au) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtMEALeyUEg7pyt2/2dsb2JhbACBW6wX X-IronPort-AV: E=Sophos;i="4.27,631,1204464600"; d="scan'208";a="124714950" Received: from ppp59-167-43-118.lns2.cbr1.internode.on.net (HELO bullseye.apana.org.au) ([59.167.43.118]) by ipmail01.adl6.internode.on.net with ESMTP; 12 Jun 2008 21:55:23 +0930 Received: from [192.168.63.10] (tenring.andymac.org [192.168.63.10]) by bullseye.apana.org.au (8.14.2/8.14.2) with ESMTP id m5CCNFeW016382; Thu, 12 Jun 2008 22:23:16 +1000 (EST) (envelope-from andymac@bullseye.andymac.org) Message-ID: <485107C2.7080202@bullseye.andymac.org> Date: Thu, 12 Jun 2008 22:25:54 +1100 From: Andrew MacIntyre User-Agent: Thunderbird 2.0.0.14 (OS/2/20080509) MIME-Version: 1.0 To: Felipe Neuwald References: <928b5da90806111738h55bbbbb0y3e9731323a1561f4@mail.gmail.com> In-Reply-To: <928b5da90806111738h55bbbbb0y3e9731323a1561f4@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, freebsd-python@freebsd.org Subject: Re: Performance with python and FreeBSD 7.0 amd64 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2008 12:40:43 -0000 Felipe Neuwald wrote: > We have a few servers running zope + plone. On one server running > FreeBSD 6.3-STABLE i386, I got no problems, but, with one server > running FreeBSD 7.0-STABLE amd64, same versions of applications, I got > some errors, like the following: > > dmesg result: > pid 74775 (python), uid 1002: exited on signal 11 segmentation violation {...} > pid 75949 (python), uid 1002: exited on signal 10 bus error {...} > pid 75999 (python), uid 1002: exited on signal 4 illegal instruction {...} Hmm... that's an interesting mix of failures. I have seen bus errors when Python runs out of stack space either in the main thread or child threads (not an unknown issue with Zope). gcc 4.x in my limited experience generates sometimes noticeably larger stack frames than gcc 3.x (which is standard on 6.x), which can provoke unexpected stack exhaustion. You don't mention whether you're using a local build or a binary package. Nor do you mention the point release (python 2.4.5 is the most recent in the 2.4 series). The default thread stack size according to my 6.3 box's ports is 1MB for Python 2.4.4) which should be adequate for most circumstances. The illegal instruction failure suggests something wrong with your binaries (including those built for Zope). The segmentation violations often indicate a problem with reference counts, frequently attributable to bugs in 3rd party extensions. You might want to check that all binaries for Python, Zope & Plone (if it has any) link against the same libraries. If you can snaffle cores, you might want to try and extract backtraces from gdb (debugging symbols would make this more productive...) -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From owner-freebsd-python@FreeBSD.ORG Thu Jun 12 16:32:25 2008 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B99D91065677 for ; Thu, 12 Jun 2008 16:32:25 +0000 (UTC) (envelope-from felipebgn@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.172]) by mx1.freebsd.org (Postfix) with ESMTP id 4B7EC8FC0C for ; Thu, 12 Jun 2008 16:32:24 +0000 (UTC) (envelope-from felipebgn@gmail.com) Received: by ug-out-1314.google.com with SMTP id q2so398726uge.37 for ; Thu, 12 Jun 2008 09:32:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=8tHB4N7dyXI05zA/eWkW5r4+nxp/1cxTiBnXWX9SY9U=; b=sZ8LVlsFYeoti2xJ15co7Z/NPmZuF/VfF3HWo4ep9yGVLtyCEuVvbVYxkR186Xk2KD eb+O8n850/yOtVqiMObMcwq0XZ7VYHwW3qbG70jSVyR+B6AgTY+nOxGI95KDJXvehk+i BkTD4LUBFQko0o+MicbpBktY7PqejmdGlYhx0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=YFyfc1XgOZUXnbtuxxuzoCINqccTF6zJfpGVl3DXQ5AWBh3bgGxbdr35tnfG04a4+5 Q/CezLDrT4swGl/4Z1/1GDRmB2Y/f5iwCuissd3xN14aU6Zcz6Jkpcbwv2+5DeGH6DNE OEG0oEQuNkELxvszjM8SpKSNfJSXQRSFnoDvw= Received: by 10.210.92.8 with SMTP id p8mr1354481ebb.182.1213288343894; Thu, 12 Jun 2008 09:32:23 -0700 (PDT) Received: by 10.210.19.16 with HTTP; Thu, 12 Jun 2008 09:32:23 -0700 (PDT) Message-ID: <928b5da90806120932v49113d35if81b12c45c86c662@mail.gmail.com> Date: Thu, 12 Jun 2008 13:32:23 -0300 From: "Felipe Neuwald" To: "Andrew MacIntyre" In-Reply-To: <485107C2.7080202@bullseye.andymac.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <928b5da90806111738h55bbbbb0y3e9731323a1561f4@mail.gmail.com> <485107C2.7080202@bullseye.andymac.org> Cc: freebsd-performance@freebsd.org, freebsd-python@freebsd.org Subject: Re: Performance with python and FreeBSD 7.0 amd64 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2008 16:32:25 -0000 > > Hmm... that's an interesting mix of failures. > > I have seen bus errors when Python runs out of stack space either in the > main thread or child threads (not an unknown issue with Zope). > > gcc 4.x in my limited experience generates sometimes noticeably larger > stack frames than gcc 3.x (which is standard on 6.x), which can provoke > unexpected stack exhaustion. > > You don't mention whether you're using a local build or a binary package. > Nor do you mention the point release (python 2.4.5 is the most recent in > the 2.4 series). I'm using a local build, installed via ports tree (python24-2.4.5_1). > The default thread stack size according to my 6.3 box's ports is 1MB for > Python 2.4.4) which should be adequate for most circumstances. > > The illegal instruction failure suggests something wrong with your > binaries (including those built for Zope). > > The segmentation violations often indicate a problem with reference > counts, frequently attributable to bugs in 3rd party extensions. > > You might want to check that all binaries for Python, Zope & Plone (if it > has any) link against the same libraries. Ok, I'll try to check these. I'm not the python + zope + plone guy, I'm the FreeBSD administrator. I'll have to work with the application team to find the solution for these problem. > If you can snaffle cores, you might want to try and extract backtraces > from gdb (debugging symbols would make this more productive...) Ok, I'll also try to get more information with cores. Thanks, Felipe Neuwald. From owner-freebsd-python@FreeBSD.ORG Thu Jun 12 17:57:07 2008 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C19A106567A for ; Thu, 12 Jun 2008 17:57:07 +0000 (UTC) (envelope-from felipebgn@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.174]) by mx1.freebsd.org (Postfix) with ESMTP id 950988FC18 for ; Thu, 12 Jun 2008 17:57:06 +0000 (UTC) (envelope-from felipebgn@gmail.com) Received: by ug-out-1314.google.com with SMTP id q2so417108uge.37 for ; Thu, 12 Jun 2008 10:57:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=V3+UDLI7uVFCcg7hazI6OpnC6r2xyaWQJZ2oRasdVPI=; b=k7MV7l1iGEnLobDDJzQpGFAdPGBhIhY0aWbct/AOZ/HkmOLbN9O+ck5rF4t++KSJ5L gL4jkZsA4Dgy3q+nuzEh8QyTDzySQh2SJ/W+iifqL2uFK2plUbses46a4v7683FewziX oOR7w0WtkoC8UXL6Jese7seWsPHwBYR+T8ISg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=ZaWThVLtv8zAjNZLUXlsnoqOMHoZnBuwhwCOA7vpk9piIJOgK9jMr2wQlS1BSEeEV5 ODuBk3I7n0fXY5BFT0sT6AzIK/JS/m8+9TaKCG8oBk/KNuc2Ynt0LRoqrh3+xfM8jvrA LJBT7plQ9QKfZX/SSGXLi1BGtJd7UQWAUIKII= Received: by 10.210.49.19 with SMTP id w19mr1515049ebw.11.1213293425231; Thu, 12 Jun 2008 10:57:05 -0700 (PDT) Received: by 10.210.19.16 with HTTP; Thu, 12 Jun 2008 10:57:05 -0700 (PDT) Message-ID: <928b5da90806121057j48c178bdp773adfe759561e15@mail.gmail.com> Date: Thu, 12 Jun 2008 14:57:05 -0300 From: "Felipe Neuwald" To: "Andrew MacIntyre" In-Reply-To: <485107C2.7080202@bullseye.andymac.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <928b5da90806111738h55bbbbb0y3e9731323a1561f4@mail.gmail.com> <485107C2.7080202@bullseye.andymac.org> Cc: freebsd-performance@freebsd.org, freebsd-python@freebsd.org Subject: Re: Performance with python and FreeBSD 7.0 amd64 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2008 17:57:07 -0000 Andrew, I'll try to recompile python with "HUGE STACK SIZE" option. Let's see. Felipe Neuwald. From owner-freebsd-python@FreeBSD.ORG Fri Jun 13 12:45:53 2008 Return-Path: Delivered-To: freebsd-python@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBD641065670 for ; Fri, 13 Jun 2008 12:45:53 +0000 (UTC) (envelope-from felipebgn@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.169]) by mx1.freebsd.org (Postfix) with ESMTP id 6F12B8FC19 for ; Fri, 13 Jun 2008 12:45:53 +0000 (UTC) (envelope-from felipebgn@gmail.com) Received: by ug-out-1314.google.com with SMTP id q2so73219uge.37 for ; Fri, 13 Jun 2008 05:45:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=hje+WtmV4sOu8lQJZPWX+viBhFAe7K/6u7D/CmfWyFI=; b=cIYhNkI4J4yc8qUlsXgxEdHdTohVkdNyN7jgqQvAAzHbAdjpnZgko0DqnAsf/KqKE7 lPa6BWYEe+vUS5pv17C3dG9Ax/YUiKVq2M2J0EAe9KloRQCg9UeG/wU3EPWPaZ3cSF3y WbYWeF7hN8jZrqAaiq3r23fXToVJxwVvxpw2I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=I+F1Ol8kuI9aH8KHcCbcbEyE5L6JFYefDo75OZITq/Nbmg127DnL4GcPS6AsrIoLsW yBiIMRXKwMXut4Y0NXx7Cx/OgY1yJ4iEvN8bSJcW/smyLsI5AuZhZaFrOvLFRtUFqyge q/t08+OslBfHOADd3dlhRRcJB0w2puIZahVsE= Received: by 10.210.58.13 with SMTP id g13mr2545601eba.54.1213361151745; Fri, 13 Jun 2008 05:45:51 -0700 (PDT) Received: by 10.210.19.16 with HTTP; Fri, 13 Jun 2008 05:45:51 -0700 (PDT) Message-ID: <928b5da90806130545w37621619t20dab5495318459a@mail.gmail.com> Date: Fri, 13 Jun 2008 09:45:51 -0300 From: "Felipe Neuwald" To: "Andrew MacIntyre" In-Reply-To: <928b5da90806121057j48c178bdp773adfe759561e15@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <928b5da90806111738h55bbbbb0y3e9731323a1561f4@mail.gmail.com> <485107C2.7080202@bullseye.andymac.org> <928b5da90806121057j48c178bdp773adfe759561e15@mail.gmail.com> Cc: freebsd-performance@freebsd.org, freebsd-python@freebsd.org Subject: Re: Performance with python and FreeBSD 7.0 amd64 X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jun 2008 12:45:54 -0000 Andrew and all, After recompile python 2.4 with HUGE_STACK_SIZE option, I got no more problems. I'll still wait the weekend to say it again, and wait for the customer reply about system performance / errors. If I got news, I'll send to you. Thank you very much, Felipe Neuwald. 2008/6/12 Felipe Neuwald : > Andrew, I'll try to recompile python with "HUGE STACK SIZE" option. Let's see. > > Felipe Neuwald. >