From owner-freebsd-ports@FreeBSD.ORG Tue Jun 10 16:45:34 2008 Return-Path: Delivered-To: ports@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37554106564A for ; Tue, 10 Jun 2008 16:45:34 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BA8FC8FC14 for ; Tue, 10 Jun 2008 16:45:31 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <484EAFAC.3020208@FreeBSD.org> Date: Tue, 10 Jun 2008 18:45:32 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: ports@FreeBSD.org Content-Type: multipart/mixed; boundary="------------040303030706050704010102" Cc: Subject: INDEX build optimizations - please review X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2008 16:45:34 -0000 This is a multi-part message in MIME format. --------------040303030706050704010102 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Please review and test the following patches that optimize port INDEX builds (and as a side-effect, other recursive tree traversals). I am particularly interested in a comparison between old and new indexes built locally: the only diff should be in audio/festvox-hvs [1]. The patches remove most of the extraneous command executions required for each invocation of bsd.port.mk (e.g. via 'make describe' in a port), by replacing external command invocations with shell or make builtins, as well as caching of constants. Another important optimization is to use /rescue/sh instead of /bin/sh for index builds. The former is statically linked and this is much faster to execute. One further change I didn't do would be to move the WWW specification from pkg-descr into a Makefile variable (this could be done mechanically). The new 'make describe' target runs entirely using shell builtins apart from the need to sed pkg-descr to extract the WWW [2] (previously 'describe' was a combination of several shell executions and a perl script, invoked for every port). With these patches an index build on an 8 core system drops from 796.486u 974.564s 5:25.14 544.7% 28+193k 37252+719io 27pf+0w to 642.846u 164.520s 2:31.29 533.6% 67+297k 0+721io 0pf+0w (or with statically linked sed: 637.805u 142.335s 2:27.32 529.5% 71+304k 0+720io 0pf+0w ) The new version requires 69301 forks and 68614 execs (would be ~19000 fewer of each without WWW in pkg-descr), compared to 252383 forks and 226875 execs with the current version. The resulting index is identical except for one port [1] but generating it is more than twice as fast (and uses 6 times less system CPU). I am only getting ~530% CPU utilization because of contention in the scheduler, so there is scope for going as low as 100 seconds if this went away. Further improvements are no doubt possible but would require profiling the work done within make(1) to see where it is spending its time (variable setting, conditional evaluation, loop invocation, regexp processing, etc). ********************************************************************** ********************* NOTE TO PORT DEVELOPERS ************************ ********************************************************************** Variable assignments with != are bad! Try as hard as you can to avoid using them -- especially in Mk/*! Every time something processes your makefile it will spawn a command, even if it is not relevant for the operation being performed. If you need to run shell commands, try to isolate them within a makefile target. You can avoid code duplication by assigning the *shell commands* (not their output) to a variable and inserting it into your code block. e.g. instead of -- VARIABLE!= do some shell stuff; do some other stuff target: echo ${VARIABLE} -- do this (or similar): -- VARIABLE_CMDS= do some shell stuff; do some other stuff target: echo $(${VARIABLE_CMDS}) -- This defers the command execution to the point where the target runs, so in the case when the target is *not* run, then you avoid wasting one or more process executions. Kris [1] This patch exposed a bug: for some reason no dependencies were previously recorded for audio/festvox-hvs! Probably because it only has RUN_DEPENDS and the 'make describe' perl script was broken. [2] Actually I am not happy with this but couldn't think of a way to do it better. Having to fork the subshell costs about 60 seconds of system time and 10 of wall time. --------------040303030706050704010102 Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0"; name="index-fast.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="index-fast.diff" Index: Makefile =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Makefile,v retrieving revision 1.103 diff -u -r1.103 Makefile --- Makefile 27 Sep 2007 05:36:26 -0000 1.103 +++ Makefile 10 Jun 2008 09:56:17 -0000 @@ -103,6 +103,7 @@ tmpdir=`/usr/bin/mktemp -d -t index` || exit 1; \ trap "rm -rf $${tmpdir}; exit 1" 1 2 3 5 10 13 15; \ ( cd ${.CURDIR} && make -j${INDEX_JOBS} INDEX_TMPDIR=$${tmpdir} BUILDING_INDEX=1 \ + __MAKE_SHELL=/rescue/sh \ ECHO_MSG="${INDEX_ECHO_MSG}" describe ) || \ (rm -rf $${tmpdir} ; \ if [ "${INDEX_QUIET}" = "" ]; then \ Index: Mk/bsd.java.mk =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.java.mk,v retrieving revision 1.82 diff -u -r1.82 bsd.java.mk --- Mk/bsd.java.mk 28 Oct 2007 15:09:43 -0000 1.82 +++ Mk/bsd.java.mk 10 Jun 2008 09:56:17 -0000 @@ -249,34 +249,30 @@ . endfor # Error checking: JAVA_VERSION +.if !defined(_JAVA_VERSION_LIST_REGEXP) _JAVA_VERSION_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_VERSION_LIST}" | ${SED} "s/ /\\\|/g" -_ERROR_CHECKING_JAVA_VERSION!= ${ECHO_CMD} "${JAVA_VERSION}" | ${TR} " " "\n" \ - | ${GREP} -v "${_JAVA_VERSION_LIST_REGEXP}" || true -. if (${_ERROR_CHECKING_JAVA_VERSION} != "") +.endif check-makevars:: - @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VERSION}\" is not a valid value for JAVA_VERSION. It should be one or more of: ${__JAVA_VERSION_LIST} (with an optional \"+\" suffix.)"; - @${FALSE} -. endif + @${ECHO_CMD} "${JAVA_VERSION}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_VERSION_LIST_REGEXP}" || \ + (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VERSION}\" is not a valid value for JAVA_VERSION. It should be one or more of: ${__JAVA_VERSION_LIST} (with an optional \"+\" suffix.)"; ${FALSE}) # Error checking: JAVA_VENDOR +.if !defined(_JAVA_VENDOR_LIST_REGEXP) _JAVA_VENDOR_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_VENDOR_LIST}" | ${SED} "s/ /\\\|/g" -_ERROR_CHECKING_JAVA_VENDOR!= ${ECHO_CMD} "${JAVA_VENDOR}" | ${TR} " " "\n" \ - | ${GREP} -v "${_JAVA_VENDOR_LIST_REGEXP}" || true -. if (${_ERROR_CHECKING_JAVA_VENDOR} != "") +.endif check-makevars:: - @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VENDOR}\" is not a valid value for JAVA_VENDOR. It should be one or more of: ${_JAVA_VENDOR_LIST}"; - @${FALSE} -. endif + @${ECHO_CMD} "${JAVA_VENDOR}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_VENDOR_LIST_REGEXP}" || \ + (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VENDOR}\" is not a valid value for JAVA_VENDOR. It should be one or more of: ${_JAVA_VENDOR_LIST}"; \ + ${FALSE}) # Error checking: JAVA_OS +.if !defined(_JAVA_OS_LIST_REGEXP) _JAVA_OS_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_OS_LIST}" | ${SED} "s/ /\\\|/g" -_ERROR_CHECKING_JAVA_OS!= ${ECHO_CMD} "${JAVA_OS}" | ${TR} " " "\n" \ - | ${GREP} -v "${_JAVA_OS_LIST_REGEXP}" || true -. if (${_ERROR_CHECKING_JAVA_OS} != "") +.endif check-makevars:: - @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_OS}\" is not a valid value for JAVA_OS. It should be one or more of: ${_JAVA_OS_LIST}"; - @${FALSE} -. endif + @${ECHO_CMD} "${JAVA_OS}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_OS_LIST_REGEXP}" || \ + (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_OS}\" is not a valid value for JAVA_OS. It should be one or more of: ${_JAVA_OS_LIST}"; \ + ${FALSE}) # Set default values for JAVA_BUILD and JAVA_RUN # When nothing is set, assume JAVA_BUILD=jdk and JAVA_RUN=jre @@ -311,20 +307,32 @@ A_JAVA_PORT_INFO:= ${A_JAVA_PORT:S/^/\${_/:S/$/_INFO}/} A_JAVA_PORT_HOME= ${A_JAVA_PORT_INFO:MHOME=*:S,HOME=,,} A_JAVA_PORT_VERSION= ${A_JAVA_PORT_INFO:MVERSION=*:C/VERSION=([0-9])\.([0-9])(.*)/\1.\2/} -A_JAVA_PORT_OS= ${A_JAVA_PORT_INFO:MOS=*:S,OS=,,} -A_JAVA_PORT_VENDOR= ${A_JAVA_PORT_INFO:MVENDOR=*:S,VENDOR=,,} +A_JAVA_PORT_OS= ${A_JAVA_PORT_INFO:MOS=*:S,OS=,,} +A_JAVA_PORT_VENDOR= ${A_JAVA_PORT_INFO:MVENDOR=*:S,VENDOR=,,} +.if !defined(_JAVA_PORTS_INSTALLED) A_JAVA_PORT_INSTALLED!= ${TEST} -x "${A_JAVA_PORT_HOME}/${_JDK_FILE}" \ && ${ECHO_CMD} "${A_JAVA_PORT}" \ || ${TRUE} __JAVA_PORTS_INSTALLED!= ${ECHO_CMD} "${__JAVA_PORTS_INSTALLED} ${A_JAVA_PORT_INSTALLED}" -A_JAVA_PORT_POSSIBLE!= ${ECHO_CMD} "${_JAVA_VERSION}" | ${GREP} -q "${A_JAVA_PORT_VERSION}" \ - && ${ECHO_CMD} "${_JAVA_OS}" | ${GREP} -q "${A_JAVA_PORT_OS}" \ - && ${ECHO_CMD} "${_JAVA_VENDOR}" | ${GREP} -q "${A_JAVA_PORT_VENDOR}" \ - && ${ECHO_CMD} "${A_JAVA_PORT}" \ - || ${TRUE} -__JAVA_PORTS_POSSIBLE!= ${ECHO_CMD} "${__JAVA_PORTS_POSSIBLE} ${A_JAVA_PORT_POSSIBLE}" +.endif + +# The magic here is that we want to test for a substring using only shell builtins (to avoid forking) +# Our shell does not have an explicit substring operator, but we can build one by using the '#' +# deletion operator ('%' would also work). We try to delete the pattern "*${substr}*" and compare it +# to the original string. If they differ, the substring matched. +# +# We can't do this in make because it doesn't allow nested modifiers ${foo:${bar}} +# +A_JAVA_PORT_POSSIBLE!= ver="${_JAVA_VERSION}"; os="${_JAVA_OS}"; vendor="${_JAVA_VENDOR}"; \ + ${TEST} "$${ver\#*${A_JAVA_PORT_VERSION}*}" != "${_JAVA_VERSION}" -a \ + "$${os\#*${A_JAVA_PORT_OS}*}" != "${_JAVA_OS}" -a \ + "$${vendor\#*${A_JAVA_PORT_VENDOR}*}" != "${_JAVA_VENDOR}" && \ + ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE} +__JAVA_PORTS_POSSIBLE:= ${__JAVA_PORTS_POSSIBLE} ${A_JAVA_PORT_POSSIBLE} . endfor +.if !defined(_JAVA_PORTS_INSTALLED) _JAVA_PORTS_INSTALLED= ${__JAVA_PORTS_INSTALLED:C/ [ ]+/ /g} +.endif _JAVA_PORTS_POSSIBLE= ${__JAVA_PORTS_POSSIBLE:C/ [ ]+/ /g} @@ -337,20 +345,28 @@ . undef _JAVA_PORTS_INSTALLED_POSSIBLE . for A_JAVA_PORT in ${_JAVA_PORTS_POSSIBLE} -A_JAVA_PORT_INSTALLED_POSSIBLE!= ${ECHO_CMD} "${_JAVA_PORTS_INSTALLED}" | ${GREP} -q "${A_JAVA_PORT}" \ - && ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE} -__JAVA_PORTS_INSTALLED_POSSIBLE!= ${ECHO_CMD} "${__JAVA_PORTS_INSTALLED_POSSIBLE} ${A_JAVA_PORT_INSTALLED_POSSIBLE}" +A_JAVA_PORT_INSTALLED_POSSIBLE!= inst="${_JAVA_PORTS_INSTALLED}"; \ + ${TEST} "$${inst\#*${A_JAVA_PORT}*}" != "${_JAVA_PORTS_INSTALLED}" && \ + ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE} +__JAVA_PORTS_INSTALLED_POSSIBLE:= ${__JAVA_PORTS_INSTALLED_POSSIBLE} ${A_JAVA_PORT_INSTALLED_POSSIBLE} . endfor -_JAVA_PORTS_INSTALLED_POSSIBLE= ${__JAVA_PORTS_INSTALLED_POSSIBLE:C/ [ ]+/ /g} +_JAVA_PORTS_INSTALLED_POSSIBLE= ${__JAVA_PORTS_INSTALLED_POSSIBLE:C/[ ]+//g} . if ${_JAVA_PORTS_INSTALLED_POSSIBLE} != "" -_JAVA_PORT!= ${ECHO_CMD} "${_JAVA_PORTS_INSTALLED_POSSIBLE}" \ - | ${AWK} '{ print $$1 }' - +. for i in ${_JAVA_PORTS_INSTALLED_POSSIBLE} +. if !defined(_JAVA_PORTS_INSTALLED_POSSIBLE_shortcircuit) +_JAVA_PORT= $i +_JAVA_PORTS_INSTALLED_POSSIBLE_shortcircuit= 1 +. endif +. endfor # If no installed JDK port fits, then pick one from the list of possible ones . else -_JAVA_PORT!= ${ECHO_CMD} "${_JAVA_PORTS_POSSIBLE}" \ - | ${AWK} '{ print $$1 }' +. for i in ${_JAVA_PORTS_POSSIBLE} +. if !defined(_JAVA_PORTS_POSSIBLE_shortcircuit) +_JAVA_PORT= $i +_JAVA_PORTS_POSSIBLE_shortcircuit= 1 +. endif +. endfor . endif _JAVA_PORT_INFO:= ${_JAVA_PORT:S/^/\${_/:S/$/_INFO}/} Index: Mk/bsd.port.mk =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.port.mk,v retrieving revision 1.593 diff -u -r1.593 bsd.port.mk --- Mk/bsd.port.mk 27 May 2008 22:12:02 -0000 1.593 +++ Mk/bsd.port.mk 10 Jun 2008 11:08:58 -0000 @@ -1235,14 +1239,13 @@ UNIQUENAME?= ${PKGNAMEPREFIX}${PORTNAME} .endif OPTIONSFILE?= ${PORT_DBDIR}/${UNIQUENAME}/options -_OPTIONSFILE!= ${ECHO_CMD} "${OPTIONSFILE}" .if defined(OPTIONS) # include OPTIONSFILE first if exists -. if exists(${_OPTIONSFILE}) && !make(rmconfig) -. include "${_OPTIONSFILE}" +. if exists(${OPTIONSFILE}) && !make(rmconfig) +. include "${OPTIONSFILE}" . endif -. if exists(${_OPTIONSFILE}.local) -. include "${_OPTIONSFILE}.local" +. if exists(${OPTIONSFILE}.local) +. include "${OPTIONSFILE}.local" . endif WITHOUT:= WITH:= @@ -1640,7 +1643,7 @@ HAVE_COMPAT_IA32_LIBS?= YES .endif .if !defined(HAVE_COMPAT_IA32_KERN) -HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -a compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi +HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -n compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi .endif .endif @@ -3265,8 +3268,8 @@ .endif .if defined(_OPTIONS_READ) @${ECHO_MSG} "===> Found saved configuration for ${_OPTIONS_READ}" -.if ${OPTIONSFILE} != ${_OPTIONSFILE} - @${ECHO_MSG} "===> *** CAUTION *** Using wrong configuration file ${_OPTIONSFILE}" +.if ${OPTIONSFILE} != ${OPTIONSFILE} + @${ECHO_MSG} "===> *** CAUTION *** Using wrong configuration file ${OPTIONSFILE}" .endif .endif @@ -5283,68 +5286,29 @@ # first to avoid gratuitous breakage. .if !target(describe) +_EXTRACT_DEPENDS=${EXTRACT_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} +_PATCH_DEPENDS=${PATCH_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} +_FETCH_DEPENDS=${FETCH_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} +_LIB_DEPENDS=${LIB_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} +_BUILD_DEPENDS=${BUILD_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} ${_LIB_DEPENDS} +_RUN_DEPENDS=${RUN_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} ${_LIB_DEPENDS} +.if exists(${DESCR}) +_DESCR=${DESCR} +.else +_DESCR=/dev/null +.endif + describe: - @${ECHO_CMD} -n "${PKGNAME}|${.CURDIR}|${PREFIX}|" -.if defined(COMMENT) - @${ECHO_CMD} -n ${COMMENT:Q} -.else - @${ECHO_CMD} -n '** No Description' -.endif - @perl -e ' \ - if ( -f q{${DESCR}} ) { \ - print q{|${DESCR}}; \ - } else { \ - print q{|/dev/null}; \ - } \ - print q{|${MAINTAINER}|${CATEGORIES}|}; \ - @edirs = map((split /:/)[1], split(q{ }, q{${EXTRACT_DEPENDS}})); \ - @pdirs = map((split /:/)[1], split(q{ }, q{${PATCH_DEPENDS}})); \ - @fdirs = map((split /:/)[1], split(q{ }, q{${FETCH_DEPENDS}})); \ - @bdirs = map((split /:/)[1], split(q{ }, q{${BUILD_DEPENDS}})); \ - @rdirs = map((split /:/)[1], split(q{ }, q{${RUN_DEPENDS}})); \ - @ldirs = map((split /:/)[1], split(q{ }, q{${LIB_DEPENDS}})); \ - for my $$i (\@edirs, \@pdirs, \@fdirs, \@bdirs, \@rdirs, \@ddirs, \@ldirs) { \ - my @dirs = @$$i; \ - @$$i = (); \ - for (@dirs) { \ - if (-d $$_) { \ - push @$$i, $$_; \ - } else { \ - print STDERR qq{${PKGNAME}: \"$$_\" non-existent -- dependency list incomplete\n}; \ - exit(1); \ - } \ - } \ - } \ - for (@edirs, @ddirs) { \ - $$xe{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xe), q{|}; \ - for (@pdirs, @ddirs) { \ - $$xp{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xp), q{|}; \ - for (@fdirs, @ddirs) { \ - $$xf{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xf), q{|}; \ - for (@bdirs, @ddirs, @ldirs) { \ - $$xb{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xb), q{|}; \ - for (@rdirs, @ddirs, @ldirs) { \ - $$xr{$$_} = 1; \ - } \ - print join(q{ }, sort keys %xr), q{|}; \ - if (open(DESCR, q{${DESCR}})) { \ - while () { \ - if (/^WWW:\s+(\S+)/) { \ - print $$1; \ - last; \ - } \ - } \ - } \ - print qq{\n};' + @${ECHO_CMD} -n "${PKGNAME}|${.CURDIR}|${PREFIX}|"; \ + ${ECHO_CMD} -n ${COMMENT:Q}; \ + ${ECHO_CMD} -n "|${_DESCR}|${MAINTAINER}|${CATEGORIES}|${_EXTRACT_DEPENDS}|${_PATCH_DEPENDS}|${_FETCH_DEPENDS}|${_BUILD_DEPENDS:O:u}|${_RUN_DEPENDS:O:u}|"; \ + set "" $$(sed -E -e '/^WWW:[[:blank:]]+/!d' -e 's,^WWW:[[:blank:]]+([[:print:]]+).*$$,\1,' ${_DESCR}); \ + echo $$2 .endif +# | (read site && ${ECHO_CMD} $${site}) || ${ECHO_CMD} +# sed -E -e '/^WWW:/!d' -e 's,^WWW:.*(http[:print:]*),\1,' ${_DESCR} | (read site && ${ECHO_CMD} $${site}) + +# ${GREP} -m1 "^WWW:" ${_DESCR} | (read foo site bar && ${ECHO_CMD} $$site) || ${ECHO_CMD} www-site: .if exists(${DESCR}) @@ -5768,23 +5732,23 @@ .if !defined(OPTIONS) @${ECHO_MSG} "===> No options to configure" .else -.if ${OPTIONSFILE} != ${_OPTIONSFILE} - @${ECHO_MSG} "===> Using wrong configuration file ${_OPTIONSFILE}" +.if ${OPTIONSFILE} != ${OPTIONSFILE} + @${ECHO_MSG} "===> Using wrong configuration file ${OPTIONSFILE}" @exit 1 .endif .if ${UID} != 0 && !defined(INSTALL_AS_USER) - @optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ + @optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ ${ECHO_MSG} "===> Switching to root credentials to create $${optionsdir}"; \ (${SU_CMD} "${SH} -c \"${MKDIR} $${optionsdir} 2> /dev/null\"") || \ (${ECHO_MSG} "===> Cannot create $${optionsdir}, check permissions"; exit 1); \ ${ECHO_MSG} "===> Returning to user credentials" .else - @(optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ + @(optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ ${MKDIR} $${optionsdir} 2> /dev/null) || \ (${ECHO_MSG} "===> Cannot create $${optionsdir}, check permissions"; exit 1) .endif - -@if [ -e ${_OPTIONSFILE} ]; then \ - . ${_OPTIONSFILE}; \ + -@if [ -e ${OPTIONSFILE} ]; then \ + . ${OPTIONSFILE}; \ fi; \ set -- ${OPTIONS} XXX; \ while [ $$# -gt 3 ]; do \ @@ -5834,11 +5798,11 @@ fi; \ done; \ if [ `${ID} -u` != 0 -a "x${INSTALL_AS_USER}" = "x" ]; then \ - ${ECHO_MSG} "===> Switching to root credentials to write ${_OPTIONSFILE}"; \ - ${SU_CMD} "${CAT} $${TMPOPTIONSFILE} > ${_OPTIONSFILE}"; \ + ${ECHO_MSG} "===> Switching to root credentials to write ${OPTIONSFILE}"; \ + ${SU_CMD} "${CAT} $${TMPOPTIONSFILE} > ${OPTIONSFILE}"; \ ${ECHO_MSG} "===> Returning to user credentials"; \ else \ - ${CAT} $${TMPOPTIONSFILE} > ${_OPTIONSFILE}; \ + ${CAT} $${TMPOPTIONSFILE} > ${OPTIONSFILE}; \ fi; \ ${RM} -f $${TMPOPTIONSFILE} .endif @@ -5855,9 +5819,9 @@ .if !target(config-conditional) config-conditional: .if defined(OPTIONS) -.if exists(${_OPTIONSFILE}) +.if exists(${OPTIONSFILE}) # scan saved options and invalidate them, if the set of options does not match - @. ${_OPTIONSFILE}; \ + @. ${OPTIONSFILE}; \ set ${OPTIONS} XXX; \ while [ $$# -gt 3 ]; do \ withvar=WITH_$$1; \ @@ -5889,8 +5853,8 @@ showconfig: .if defined(OPTIONS) @${ECHO_MSG} "===> The following configuration options are available for ${PKGNAME}:" - -@if [ -e ${_OPTIONSFILE} ]; then \ - . ${_OPTIONSFILE}; \ + -@if [ -e ${OPTIONSFILE} ]; then \ + . ${OPTIONSFILE}; \ fi; \ set -- ${OPTIONS} XXX; \ while [ $$# -gt 3 ]; do \ @@ -5915,16 +5879,16 @@ .if !target(rmconfig) rmconfig: -.if defined(OPTIONS) && exists(${_OPTIONSFILE}) +.if defined(OPTIONS) && exists(${OPTIONSFILE}) -@${ECHO_MSG} "===> Removing user-configured options for ${PKGNAME}"; \ - optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ + optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \ if [ `${ID} -u` != 0 -a "x${INSTALL_AS_USER}" = "x" ]; then \ - ${ECHO_MSG} "===> Switching to root credentials to remove ${_OPTIONSFILE} and $${optionsdir}"; \ - ${SU_CMD} "${RM} -f ${_OPTIONSFILE} ; \ + ${ECHO_MSG} "===> Switching to root credentials to remove ${OPTIONSFILE} and $${optionsdir}"; \ + ${SU_CMD} "${RM} -f ${OPTIONSFILE} ; \ ${RMDIR} $${optionsdir}"; \ ${ECHO_MSG} "===> Returning to user credentials"; \ else \ - ${RM} -f ${_OPTIONSFILE}; \ + ${RM} -f ${OPTIONSFILE}; \ ${RMDIR} $${optionsdir}; \ fi .else Index: Mk/bsd.port.subdir.mk =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.port.subdir.mk,v retrieving revision 1.74 diff -u -r1.74 bsd.port.subdir.mk --- Mk/bsd.port.subdir.mk 12 Mar 2008 00:13:06 -0000 1.74 +++ Mk/bsd.port.subdir.mk 10 Jun 2008 11:59:30 -0000 @@ -63,13 +63,15 @@ STRIP?= -s .endif +# These are variables that are invariant for the lifetime of a recursive port traversal +# (index build, etc), so it is more efficient to precompute them here and pass them in +# to child makes explicitly, instead of recomputing them tens of thousands of times. + .if !defined(NOPRECIOUSMAKEVARS) .if !defined(ARCH) ARCH!= ${UNAME} -p .endif -.if !defined(OSREL) -OSREL!= ${UNAME} -r | ${SED} -e 's/[-(].*//' -.endif + .if !defined(OSVERSION) .if exists(/usr/include/sys/param.h) OSVERSION!= ${AWK} '/^\#define[[:blank:]]__FreeBSD_version/ {print $$3}' < /usr/include/sys/param.h @@ -79,23 +81,76 @@ OSVERSION!= ${SYSCTL} -n kern.osreldate .endif .endif + +.if !defined(_OSRELEASE) +_OSRELEASE!= uname -r +.endif +.if !defined(OSREL) +OSREL= ${_OSRELEASE:C/[-(].*//} .endif -INDEXDIR?= ${PORTSDIR} -INDEXFILE?= INDEX-${OSVERSION:C/([0-9]).*/\1/} +.if !defined(OPSYS) +OPSYS!= ${UNAME} -s +.endif + +.if ${ARCH} == "amd64" || ${ARCH} =="ia64" +.if !defined(HAVE_COMPAT_IA32_KERN) +HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -n compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi +.endif +.endif +.if !defined(CONFIGURE_MAX_CMD_LEN) +CONFIGURE_MAX_CMD_LEN!= ${SYSCTL} -n kern.argmax +.endif + +.if !defined(PYTHON_DEFAULT_VERSION) +PYTHON_DEFAULT_VERSION!= make -V PYTHON_DEFAULT_VERSION USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(PYTHON_DEFAULT_PORTVERSION) +# We are caching the PYTHON_PORTVERSION of the default python version so we can reuse it in the +# common case. +PYTHON_DEFAULT_PORTVERSION!= make -V PYTHON_PORTVERSION USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(PYTHONBASE) +PYTHONBASE!= make -V PYTHONBASE USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(_JAVA_VERSION_LIST_REGEXP) +_JAVA_VERSION_LIST_REGEXP!= make -V _JAVA_VERSION_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(_JAVA_VENDOR_LIST_REGEXP) +_JAVA_VENDOR_LIST_REGEXP!= make -V _JAVA_VENDOR_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(_JAVA_OS_LIST_REGEXP) +_JAVA_OS_LIST_REGEXP!= make -V _JAVA_OS_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(_JAVA_PORTS_INSTALLED) +_JAVA_PORTS_INSTALLED!= make -V _JAVA_PORTS_INSTALLED USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk +.endif + +.if !defined(UID) UID!= ${ID} -u +.endif + .if exists(${LOCALBASE}/sbin/pkg_info) PKG_INFO?= ${LOCALBASE}/sbin/pkg_info .else PKG_INFO?= /usr/sbin/pkg_info .endif +.if !defined(PKGINSTALLVER) PKGINSTALLVER!= ${PKG_INFO} -P 2>/dev/null | ${SED} -e 's/.*: //' +.endif -.if !defined(OPSYS) -OPSYS!= ${UNAME} -s .endif +INDEXDIR?= ${PORTSDIR} +INDEXFILE?= INDEX-${OSVERSION:C/([0-9]).*/\1/} + # local customization of the ports tree .if exists(${.CURDIR}/Makefile.local) .include "${.CURDIR}/Makefile.local" @@ -307,6 +362,8 @@ > $@ @${RM} -f $@.tmp $@.tmp2 $@.tmp3 $@.tmp4 +# Pass in the cached invariant variables to child makes. +# XXX Why are we trying to escape these characters using regexps and not using ':Q'? .if !defined(NOPRECIOUSMAKEVARS) .MAKEFLAGS: \ ARCH="${ARCH:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ @@ -314,7 +371,16 @@ OSREL="${OSREL:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ OSVERSION="${OSVERSION:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ UID="${UID:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ - PKGINSTALLVER="${PKGINSTALLVER:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" + PKGINSTALLVER="${PKGINSTALLVER:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \ + HAVE_COMPAT_IA32_KERN="${HAVE_COMPAT_IA32_KERN}" \ + CONFIGURE_MAX_CMD_LEN="${CONFIGURE_MAX_CMD_LEN}" \ + PYTHON_DEFAULT_VERSION="${PYTHON_DEFAULT_VERSION}" \ + PYTHON_DEFAULT_PORTVERSION="${PYTHON_DEFAULT_PORTVERSION}" \ + PYTHONBASE="${PYTHONBASE}" \ + _JAVA_VERSION_LIST_REGEXP="${_JAVA_VERSION_LIST_REGEXP}" \ + _JAVA_VENDOR_LIST_REGEXP="${_JAVA_VENDOR_LIST_REGEXP}" \ + _JAVA_OS_LIST_REGEXP="${_JAVA_OS_LIST_REGEXP}" \ + _JAVA_PORTS_INSTALLED="${_JAVA_PORTS_INSTALLED}" .endif PORTSEARCH_DISPLAY_FIELDS?=name,path,info,maint,index,bdeps,rdeps,www Index: Mk/bsd.python.mk =================================================================== RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.python.mk,v retrieving revision 1.104 diff -u -r1.104 bsd.python.mk --- Mk/bsd.python.mk 30 May 2008 05:50:03 -0000 1.104 +++ Mk/bsd.python.mk 10 Jun 2008 12:08:49 -0000 @@ -352,14 +352,24 @@ PYTHON_VERSION?= python${_PYTHON_VERSION} PYTHON_CMD?= ${_PYTHON_CMD} +.if !defined(PYTHONBASE) PYTHONBASE!= (${PYTHON_CMD} -c 'import sys; print sys.prefix' \ 2> /dev/null || ${ECHO_CMD} ${LOCALBASE}) | ${TAIL} -1 +.endif DEPENDS_ARGS+= PYTHON_VERSION=${PYTHON_VERSION} + +# We can only use the cached version if we are using the default python version. Otherwise it +# should point to some other version we have installed, according to the port USE_PYTHON +# specification +.if !defined(PYTHON_DEFAULT_PORTVERSION) || (${PYTHON_VERSION} != ${PYTHON_DEFAULT_VERSION}) _PYTHON_PORTVERSION!= (${PYTHON_CMD} -c 'import string, sys; \ print string.split(sys.version)[0].replace("b",".b")' 2> /dev/null) | ${TAIL} -1 .if !defined(PYTHON_NO_DEPENDS) && !empty(_PYTHON_PORTVERSION) PYTHON_PORTVERSION= ${_PYTHON_PORTVERSION} .endif +.elif defined(PYTHON_DEFAULT_PORTVERSION) +PYTHON_PORTVERSION= ${PYTHON_DEFAULT_PORTVERSION} +.endif # Propagate the chosen python version to submakes. .MAKEFLAGS: PYTHON_VERSION=python${_PYTHON_VERSION} @@ -440,7 +450,9 @@ .endif .if defined(PYEASYINSTALL_ARCHDEP) +.if !defined(_OSRELEASE) _OSRELEASE!= ${UNAME} -r +.endif PYEASYINSTALL_OSARCH?= -${OPSYS:L}-${_OSRELEASE}-${ARCH} .endif PYEASYINSTALL_EGG?= ${PYDISTUTILS_PKGNAME:C/[^A-Za-z0-9.]+/_/g}-${PYDISTUTILS_PKGVERSION:C/[^A-Za-z0-9.]+/_/g}-${PYTHON_VERSION:S/thon//}${PYEASYINSTALL_OSARCH}.egg --------------040303030706050704010102--