Date: Tue, 10 Jun 2008 18:45:32 +0200 From: Kris Kennaway <kris@FreeBSD.org> To: ports@FreeBSD.org Subject: INDEX build optimizations - please review Message-ID: <484EAFAC.3020208@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------040303030706050704010102
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Please review and test the following patches that optimize port INDEX
builds (and as a side-effect, other recursive tree traversals). I am
particularly interested in a comparison between old and new indexes
built locally: the only diff should be in audio/festvox-hvs [1].
The patches remove most of the extraneous command executions required
for each invocation of bsd.port.mk (e.g. via 'make describe' in a port),
by replacing external command invocations with shell or make builtins,
as well as caching of constants.
Another important optimization is to use /rescue/sh instead of /bin/sh
for index builds. The former is statically linked and this is much
faster to execute.
One further change I didn't do would be to move the WWW specification
from pkg-descr into a Makefile variable (this could be done
mechanically). The new 'make describe' target runs entirely using shell
builtins apart from the need to sed pkg-descr to extract the WWW [2]
(previously 'describe' was a combination of several shell executions and
a perl script, invoked for every port).
With these patches an index build on an 8 core system drops from
796.486u 974.564s 5:25.14 544.7% 28+193k 37252+719io 27pf+0w
to
642.846u 164.520s 2:31.29 533.6% 67+297k 0+721io 0pf+0w
(or with statically linked sed:
637.805u 142.335s 2:27.32 529.5% 71+304k 0+720io 0pf+0w
)
The new version requires 69301 forks and 68614 execs (would be ~19000
fewer of each without WWW in pkg-descr), compared to 252383 forks and
226875 execs with the current version.
The resulting index is identical except for one port [1] but generating
it is more than twice as fast (and uses 6 times less system CPU).
I am only getting ~530% CPU utilization because of contention in the
scheduler, so there is scope for going as low as 100 seconds if this
went away. Further improvements are no doubt possible but would require
profiling the work done within make(1) to see where it is spending its
time (variable setting, conditional evaluation, loop invocation, regexp
processing, etc).
**********************************************************************
********************* NOTE TO PORT DEVELOPERS ************************
**********************************************************************
Variable assignments with != are bad! Try as hard as you can to avoid
using them -- especially in Mk/*! Every time something processes your
makefile it will spawn a command, even if it is not relevant for the
operation being performed. If you need to run shell commands, try to
isolate them within a makefile target. You can avoid code duplication
by assigning the *shell commands* (not their output) to a variable and
inserting it into your code block.
e.g. instead of
--
VARIABLE!= do some shell stuff; do some other stuff
target:
echo ${VARIABLE}
--
do this (or similar):
--
VARIABLE_CMDS= do some shell stuff; do some other stuff
target:
echo $(${VARIABLE_CMDS})
--
This defers the command execution to the point where the target runs, so
in the case when the target is *not* run, then you avoid wasting one or
more process executions.
Kris
[1] This patch exposed a bug: for some reason no dependencies were
previously recorded for audio/festvox-hvs! Probably because it only has
RUN_DEPENDS and the 'make describe' perl script was broken.
[2] Actually I am not happy with this but couldn't think of a way to do
it better. Having to fork the subshell costs about 60 seconds of system
time and 10 of wall time.
--------------040303030706050704010102
Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0";
name="index-fast.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="index-fast.diff"
Index: Makefile
===================================================================
RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Makefile,v
retrieving revision 1.103
diff -u -r1.103 Makefile
--- Makefile 27 Sep 2007 05:36:26 -0000 1.103
+++ Makefile 10 Jun 2008 09:56:17 -0000
@@ -103,6 +103,7 @@
tmpdir=`/usr/bin/mktemp -d -t index` || exit 1; \
trap "rm -rf $${tmpdir}; exit 1" 1 2 3 5 10 13 15; \
( cd ${.CURDIR} && make -j${INDEX_JOBS} INDEX_TMPDIR=$${tmpdir} BUILDING_INDEX=1 \
+ __MAKE_SHELL=/rescue/sh \
ECHO_MSG="${INDEX_ECHO_MSG}" describe ) || \
(rm -rf $${tmpdir} ; \
if [ "${INDEX_QUIET}" = "" ]; then \
Index: Mk/bsd.java.mk
===================================================================
RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.java.mk,v
retrieving revision 1.82
diff -u -r1.82 bsd.java.mk
--- Mk/bsd.java.mk 28 Oct 2007 15:09:43 -0000 1.82
+++ Mk/bsd.java.mk 10 Jun 2008 09:56:17 -0000
@@ -249,34 +249,30 @@
. endfor
# Error checking: JAVA_VERSION
+.if !defined(_JAVA_VERSION_LIST_REGEXP)
_JAVA_VERSION_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_VERSION_LIST}" | ${SED} "s/ /\\\|/g"
-_ERROR_CHECKING_JAVA_VERSION!= ${ECHO_CMD} "${JAVA_VERSION}" | ${TR} " " "\n" \
- | ${GREP} -v "${_JAVA_VERSION_LIST_REGEXP}" || true
-. if (${_ERROR_CHECKING_JAVA_VERSION} != "")
+.endif
check-makevars::
- @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VERSION}\" is not a valid value for JAVA_VERSION. It should be one or more of: ${__JAVA_VERSION_LIST} (with an optional \"+\" suffix.)";
- @${FALSE}
-. endif
+ @${ECHO_CMD} "${JAVA_VERSION}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_VERSION_LIST_REGEXP}" || \
+ (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VERSION}\" is not a valid value for JAVA_VERSION. It should be one or more of: ${__JAVA_VERSION_LIST} (with an optional \"+\" suffix.)"; ${FALSE})
# Error checking: JAVA_VENDOR
+.if !defined(_JAVA_VENDOR_LIST_REGEXP)
_JAVA_VENDOR_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_VENDOR_LIST}" | ${SED} "s/ /\\\|/g"
-_ERROR_CHECKING_JAVA_VENDOR!= ${ECHO_CMD} "${JAVA_VENDOR}" | ${TR} " " "\n" \
- | ${GREP} -v "${_JAVA_VENDOR_LIST_REGEXP}" || true
-. if (${_ERROR_CHECKING_JAVA_VENDOR} != "")
+.endif
check-makevars::
- @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VENDOR}\" is not a valid value for JAVA_VENDOR. It should be one or more of: ${_JAVA_VENDOR_LIST}";
- @${FALSE}
-. endif
+ @${ECHO_CMD} "${JAVA_VENDOR}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_VENDOR_LIST_REGEXP}" || \
+ (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_VENDOR}\" is not a valid value for JAVA_VENDOR. It should be one or more of: ${_JAVA_VENDOR_LIST}"; \
+ ${FALSE})
# Error checking: JAVA_OS
+.if !defined(_JAVA_OS_LIST_REGEXP)
_JAVA_OS_LIST_REGEXP!= ${ECHO_CMD} "${_JAVA_OS_LIST}" | ${SED} "s/ /\\\|/g"
-_ERROR_CHECKING_JAVA_OS!= ${ECHO_CMD} "${JAVA_OS}" | ${TR} " " "\n" \
- | ${GREP} -v "${_JAVA_OS_LIST_REGEXP}" || true
-. if (${_ERROR_CHECKING_JAVA_OS} != "")
+.endif
check-makevars::
- @${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_OS}\" is not a valid value for JAVA_OS. It should be one or more of: ${_JAVA_OS_LIST}";
- @${FALSE}
-. endif
+ @${ECHO_CMD} "${JAVA_OS}" | ${TR} " " "\n" | ${GREP} -q "${_JAVA_OS_LIST_REGEXP}" || \
+ (${ECHO_CMD} "${PKGNAME}: Makefile error: \"${JAVA_OS}\" is not a valid value for JAVA_OS. It should be one or more of: ${_JAVA_OS_LIST}"; \
+ ${FALSE})
# Set default values for JAVA_BUILD and JAVA_RUN
# When nothing is set, assume JAVA_BUILD=jdk and JAVA_RUN=jre
@@ -311,20 +307,32 @@
A_JAVA_PORT_INFO:= ${A_JAVA_PORT:S/^/\${_/:S/$/_INFO}/}
A_JAVA_PORT_HOME= ${A_JAVA_PORT_INFO:MHOME=*:S,HOME=,,}
A_JAVA_PORT_VERSION= ${A_JAVA_PORT_INFO:MVERSION=*:C/VERSION=([0-9])\.([0-9])(.*)/\1.\2/}
-A_JAVA_PORT_OS= ${A_JAVA_PORT_INFO:MOS=*:S,OS=,,}
-A_JAVA_PORT_VENDOR= ${A_JAVA_PORT_INFO:MVENDOR=*:S,VENDOR=,,}
+A_JAVA_PORT_OS= ${A_JAVA_PORT_INFO:MOS=*:S,OS=,,}
+A_JAVA_PORT_VENDOR= ${A_JAVA_PORT_INFO:MVENDOR=*:S,VENDOR=,,}
+.if !defined(_JAVA_PORTS_INSTALLED)
A_JAVA_PORT_INSTALLED!= ${TEST} -x "${A_JAVA_PORT_HOME}/${_JDK_FILE}" \
&& ${ECHO_CMD} "${A_JAVA_PORT}" \
|| ${TRUE}
__JAVA_PORTS_INSTALLED!= ${ECHO_CMD} "${__JAVA_PORTS_INSTALLED} ${A_JAVA_PORT_INSTALLED}"
-A_JAVA_PORT_POSSIBLE!= ${ECHO_CMD} "${_JAVA_VERSION}" | ${GREP} -q "${A_JAVA_PORT_VERSION}" \
- && ${ECHO_CMD} "${_JAVA_OS}" | ${GREP} -q "${A_JAVA_PORT_OS}" \
- && ${ECHO_CMD} "${_JAVA_VENDOR}" | ${GREP} -q "${A_JAVA_PORT_VENDOR}" \
- && ${ECHO_CMD} "${A_JAVA_PORT}" \
- || ${TRUE}
-__JAVA_PORTS_POSSIBLE!= ${ECHO_CMD} "${__JAVA_PORTS_POSSIBLE} ${A_JAVA_PORT_POSSIBLE}"
+.endif
+
+# The magic here is that we want to test for a substring using only shell builtins (to avoid forking)
+# Our shell does not have an explicit substring operator, but we can build one by using the '#'
+# deletion operator ('%' would also work). We try to delete the pattern "*${substr}*" and compare it
+# to the original string. If they differ, the substring matched.
+#
+# We can't do this in make because it doesn't allow nested modifiers ${foo:${bar}}
+#
+A_JAVA_PORT_POSSIBLE!= ver="${_JAVA_VERSION}"; os="${_JAVA_OS}"; vendor="${_JAVA_VENDOR}"; \
+ ${TEST} "$${ver\#*${A_JAVA_PORT_VERSION}*}" != "${_JAVA_VERSION}" -a \
+ "$${os\#*${A_JAVA_PORT_OS}*}" != "${_JAVA_OS}" -a \
+ "$${vendor\#*${A_JAVA_PORT_VENDOR}*}" != "${_JAVA_VENDOR}" && \
+ ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE}
+__JAVA_PORTS_POSSIBLE:= ${__JAVA_PORTS_POSSIBLE} ${A_JAVA_PORT_POSSIBLE}
. endfor
+.if !defined(_JAVA_PORTS_INSTALLED)
_JAVA_PORTS_INSTALLED= ${__JAVA_PORTS_INSTALLED:C/ [ ]+/ /g}
+.endif
_JAVA_PORTS_POSSIBLE= ${__JAVA_PORTS_POSSIBLE:C/ [ ]+/ /g}
@@ -337,20 +345,28 @@
. undef _JAVA_PORTS_INSTALLED_POSSIBLE
. for A_JAVA_PORT in ${_JAVA_PORTS_POSSIBLE}
-A_JAVA_PORT_INSTALLED_POSSIBLE!= ${ECHO_CMD} "${_JAVA_PORTS_INSTALLED}" | ${GREP} -q "${A_JAVA_PORT}" \
- && ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE}
-__JAVA_PORTS_INSTALLED_POSSIBLE!= ${ECHO_CMD} "${__JAVA_PORTS_INSTALLED_POSSIBLE} ${A_JAVA_PORT_INSTALLED_POSSIBLE}"
+A_JAVA_PORT_INSTALLED_POSSIBLE!= inst="${_JAVA_PORTS_INSTALLED}"; \
+ ${TEST} "$${inst\#*${A_JAVA_PORT}*}" != "${_JAVA_PORTS_INSTALLED}" && \
+ ${ECHO_CMD} "${A_JAVA_PORT}" || ${TRUE}
+__JAVA_PORTS_INSTALLED_POSSIBLE:= ${__JAVA_PORTS_INSTALLED_POSSIBLE} ${A_JAVA_PORT_INSTALLED_POSSIBLE}
. endfor
-_JAVA_PORTS_INSTALLED_POSSIBLE= ${__JAVA_PORTS_INSTALLED_POSSIBLE:C/ [ ]+/ /g}
+_JAVA_PORTS_INSTALLED_POSSIBLE= ${__JAVA_PORTS_INSTALLED_POSSIBLE:C/[ ]+//g}
. if ${_JAVA_PORTS_INSTALLED_POSSIBLE} != ""
-_JAVA_PORT!= ${ECHO_CMD} "${_JAVA_PORTS_INSTALLED_POSSIBLE}" \
- | ${AWK} '{ print $$1 }'
-
+. for i in ${_JAVA_PORTS_INSTALLED_POSSIBLE}
+. if !defined(_JAVA_PORTS_INSTALLED_POSSIBLE_shortcircuit)
+_JAVA_PORT= $i
+_JAVA_PORTS_INSTALLED_POSSIBLE_shortcircuit= 1
+. endif
+. endfor
# If no installed JDK port fits, then pick one from the list of possible ones
. else
-_JAVA_PORT!= ${ECHO_CMD} "${_JAVA_PORTS_POSSIBLE}" \
- | ${AWK} '{ print $$1 }'
+. for i in ${_JAVA_PORTS_POSSIBLE}
+. if !defined(_JAVA_PORTS_POSSIBLE_shortcircuit)
+_JAVA_PORT= $i
+_JAVA_PORTS_POSSIBLE_shortcircuit= 1
+. endif
+. endfor
. endif
_JAVA_PORT_INFO:= ${_JAVA_PORT:S/^/\${_/:S/$/_INFO}/}
Index: Mk/bsd.port.mk
===================================================================
RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.port.mk,v
retrieving revision 1.593
diff -u -r1.593 bsd.port.mk
--- Mk/bsd.port.mk 27 May 2008 22:12:02 -0000 1.593
+++ Mk/bsd.port.mk 10 Jun 2008 11:08:58 -0000
@@ -1235,14 +1239,13 @@
UNIQUENAME?= ${PKGNAMEPREFIX}${PORTNAME}
.endif
OPTIONSFILE?= ${PORT_DBDIR}/${UNIQUENAME}/options
-_OPTIONSFILE!= ${ECHO_CMD} "${OPTIONSFILE}"
.if defined(OPTIONS)
# include OPTIONSFILE first if exists
-. if exists(${_OPTIONSFILE}) && !make(rmconfig)
-. include "${_OPTIONSFILE}"
+. if exists(${OPTIONSFILE}) && !make(rmconfig)
+. include "${OPTIONSFILE}"
. endif
-. if exists(${_OPTIONSFILE}.local)
-. include "${_OPTIONSFILE}.local"
+. if exists(${OPTIONSFILE}.local)
+. include "${OPTIONSFILE}.local"
. endif
WITHOUT:=
WITH:=
@@ -1640,7 +1643,7 @@
HAVE_COMPAT_IA32_LIBS?= YES
.endif
.if !defined(HAVE_COMPAT_IA32_KERN)
-HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -a compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi
+HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -n compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi
.endif
.endif
@@ -3265,8 +3268,8 @@
.endif
.if defined(_OPTIONS_READ)
@${ECHO_MSG} "===> Found saved configuration for ${_OPTIONS_READ}"
-.if ${OPTIONSFILE} != ${_OPTIONSFILE}
- @${ECHO_MSG} "===> *** CAUTION *** Using wrong configuration file ${_OPTIONSFILE}"
+.if ${OPTIONSFILE} != ${OPTIONSFILE}
+ @${ECHO_MSG} "===> *** CAUTION *** Using wrong configuration file ${OPTIONSFILE}"
.endif
.endif
@@ -5283,68 +5286,29 @@
# first to avoid gratuitous breakage.
.if !target(describe)
+_EXTRACT_DEPENDS=${EXTRACT_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u}
+_PATCH_DEPENDS=${PATCH_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u}
+_FETCH_DEPENDS=${FETCH_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u}
+_LIB_DEPENDS=${LIB_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u}
+_BUILD_DEPENDS=${BUILD_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} ${_LIB_DEPENDS}
+_RUN_DEPENDS=${RUN_DEPENDS:C/^[^ :]+:([^ :]+)(:[^ :]+)?/\1/:O:u} ${_LIB_DEPENDS}
+.if exists(${DESCR})
+_DESCR=${DESCR}
+.else
+_DESCR=/dev/null
+.endif
+
describe:
- @${ECHO_CMD} -n "${PKGNAME}|${.CURDIR}|${PREFIX}|"
-.if defined(COMMENT)
- @${ECHO_CMD} -n ${COMMENT:Q}
-.else
- @${ECHO_CMD} -n '** No Description'
-.endif
- @perl -e ' \
- if ( -f q{${DESCR}} ) { \
- print q{|${DESCR}}; \
- } else { \
- print q{|/dev/null}; \
- } \
- print q{|${MAINTAINER}|${CATEGORIES}|}; \
- @edirs = map((split /:/)[1], split(q{ }, q{${EXTRACT_DEPENDS}})); \
- @pdirs = map((split /:/)[1], split(q{ }, q{${PATCH_DEPENDS}})); \
- @fdirs = map((split /:/)[1], split(q{ }, q{${FETCH_DEPENDS}})); \
- @bdirs = map((split /:/)[1], split(q{ }, q{${BUILD_DEPENDS}})); \
- @rdirs = map((split /:/)[1], split(q{ }, q{${RUN_DEPENDS}})); \
- @ldirs = map((split /:/)[1], split(q{ }, q{${LIB_DEPENDS}})); \
- for my $$i (\@edirs, \@pdirs, \@fdirs, \@bdirs, \@rdirs, \@ddirs, \@ldirs) { \
- my @dirs = @$$i; \
- @$$i = (); \
- for (@dirs) { \
- if (-d $$_) { \
- push @$$i, $$_; \
- } else { \
- print STDERR qq{${PKGNAME}: \"$$_\" non-existent -- dependency list incomplete\n}; \
- exit(1); \
- } \
- } \
- } \
- for (@edirs, @ddirs) { \
- $$xe{$$_} = 1; \
- } \
- print join(q{ }, sort keys %xe), q{|}; \
- for (@pdirs, @ddirs) { \
- $$xp{$$_} = 1; \
- } \
- print join(q{ }, sort keys %xp), q{|}; \
- for (@fdirs, @ddirs) { \
- $$xf{$$_} = 1; \
- } \
- print join(q{ }, sort keys %xf), q{|}; \
- for (@bdirs, @ddirs, @ldirs) { \
- $$xb{$$_} = 1; \
- } \
- print join(q{ }, sort keys %xb), q{|}; \
- for (@rdirs, @ddirs, @ldirs) { \
- $$xr{$$_} = 1; \
- } \
- print join(q{ }, sort keys %xr), q{|}; \
- if (open(DESCR, q{${DESCR}})) { \
- while (<DESCR>) { \
- if (/^WWW:\s+(\S+)/) { \
- print $$1; \
- last; \
- } \
- } \
- } \
- print qq{\n};'
+ @${ECHO_CMD} -n "${PKGNAME}|${.CURDIR}|${PREFIX}|"; \
+ ${ECHO_CMD} -n ${COMMENT:Q}; \
+ ${ECHO_CMD} -n "|${_DESCR}|${MAINTAINER}|${CATEGORIES}|${_EXTRACT_DEPENDS}|${_PATCH_DEPENDS}|${_FETCH_DEPENDS}|${_BUILD_DEPENDS:O:u}|${_RUN_DEPENDS:O:u}|"; \
+ set "" $$(sed -E -e '/^WWW:[[:blank:]]+/!d' -e 's,^WWW:[[:blank:]]+([[:print:]]+).*$$,\1,' ${_DESCR}); \
+ echo $$2
.endif
+# | (read site && ${ECHO_CMD} $${site}) || ${ECHO_CMD}
+# sed -E -e '/^WWW:/!d' -e 's,^WWW:.*(http[:print:]*),\1,' ${_DESCR} | (read site && ${ECHO_CMD} $${site})
+
+# ${GREP} -m1 "^WWW:" ${_DESCR} | (read foo site bar && ${ECHO_CMD} $$site) || ${ECHO_CMD}
www-site:
.if exists(${DESCR})
@@ -5768,23 +5732,23 @@
.if !defined(OPTIONS)
@${ECHO_MSG} "===> No options to configure"
.else
-.if ${OPTIONSFILE} != ${_OPTIONSFILE}
- @${ECHO_MSG} "===> Using wrong configuration file ${_OPTIONSFILE}"
+.if ${OPTIONSFILE} != ${OPTIONSFILE}
+ @${ECHO_MSG} "===> Using wrong configuration file ${OPTIONSFILE}"
@exit 1
.endif
.if ${UID} != 0 && !defined(INSTALL_AS_USER)
- @optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \
+ @optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \
${ECHO_MSG} "===> Switching to root credentials to create $${optionsdir}"; \
(${SU_CMD} "${SH} -c \"${MKDIR} $${optionsdir} 2> /dev/null\"") || \
(${ECHO_MSG} "===> Cannot create $${optionsdir}, check permissions"; exit 1); \
${ECHO_MSG} "===> Returning to user credentials"
.else
- @(optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \
+ @(optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \
${MKDIR} $${optionsdir} 2> /dev/null) || \
(${ECHO_MSG} "===> Cannot create $${optionsdir}, check permissions"; exit 1)
.endif
- -@if [ -e ${_OPTIONSFILE} ]; then \
- . ${_OPTIONSFILE}; \
+ -@if [ -e ${OPTIONSFILE} ]; then \
+ . ${OPTIONSFILE}; \
fi; \
set -- ${OPTIONS} XXX; \
while [ $$# -gt 3 ]; do \
@@ -5834,11 +5798,11 @@
fi; \
done; \
if [ `${ID} -u` != 0 -a "x${INSTALL_AS_USER}" = "x" ]; then \
- ${ECHO_MSG} "===> Switching to root credentials to write ${_OPTIONSFILE}"; \
- ${SU_CMD} "${CAT} $${TMPOPTIONSFILE} > ${_OPTIONSFILE}"; \
+ ${ECHO_MSG} "===> Switching to root credentials to write ${OPTIONSFILE}"; \
+ ${SU_CMD} "${CAT} $${TMPOPTIONSFILE} > ${OPTIONSFILE}"; \
${ECHO_MSG} "===> Returning to user credentials"; \
else \
- ${CAT} $${TMPOPTIONSFILE} > ${_OPTIONSFILE}; \
+ ${CAT} $${TMPOPTIONSFILE} > ${OPTIONSFILE}; \
fi; \
${RM} -f $${TMPOPTIONSFILE}
.endif
@@ -5855,9 +5819,9 @@
.if !target(config-conditional)
config-conditional:
.if defined(OPTIONS)
-.if exists(${_OPTIONSFILE})
+.if exists(${OPTIONSFILE})
# scan saved options and invalidate them, if the set of options does not match
- @. ${_OPTIONSFILE}; \
+ @. ${OPTIONSFILE}; \
set ${OPTIONS} XXX; \
while [ $$# -gt 3 ]; do \
withvar=WITH_$$1; \
@@ -5889,8 +5853,8 @@
showconfig:
.if defined(OPTIONS)
@${ECHO_MSG} "===> The following configuration options are available for ${PKGNAME}:"
- -@if [ -e ${_OPTIONSFILE} ]; then \
- . ${_OPTIONSFILE}; \
+ -@if [ -e ${OPTIONSFILE} ]; then \
+ . ${OPTIONSFILE}; \
fi; \
set -- ${OPTIONS} XXX; \
while [ $$# -gt 3 ]; do \
@@ -5915,16 +5879,16 @@
.if !target(rmconfig)
rmconfig:
-.if defined(OPTIONS) && exists(${_OPTIONSFILE})
+.if defined(OPTIONS) && exists(${OPTIONSFILE})
-@${ECHO_MSG} "===> Removing user-configured options for ${PKGNAME}"; \
- optionsdir=${_OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \
+ optionsdir=${OPTIONSFILE}; optionsdir=$${optionsdir%/*}; \
if [ `${ID} -u` != 0 -a "x${INSTALL_AS_USER}" = "x" ]; then \
- ${ECHO_MSG} "===> Switching to root credentials to remove ${_OPTIONSFILE} and $${optionsdir}"; \
- ${SU_CMD} "${RM} -f ${_OPTIONSFILE} ; \
+ ${ECHO_MSG} "===> Switching to root credentials to remove ${OPTIONSFILE} and $${optionsdir}"; \
+ ${SU_CMD} "${RM} -f ${OPTIONSFILE} ; \
${RMDIR} $${optionsdir}"; \
${ECHO_MSG} "===> Returning to user credentials"; \
else \
- ${RM} -f ${_OPTIONSFILE}; \
+ ${RM} -f ${OPTIONSFILE}; \
${RMDIR} $${optionsdir}; \
fi
.else
Index: Mk/bsd.port.subdir.mk
===================================================================
RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.port.subdir.mk,v
retrieving revision 1.74
diff -u -r1.74 bsd.port.subdir.mk
--- Mk/bsd.port.subdir.mk 12 Mar 2008 00:13:06 -0000 1.74
+++ Mk/bsd.port.subdir.mk 10 Jun 2008 11:59:30 -0000
@@ -63,13 +63,15 @@
STRIP?= -s
.endif
+# These are variables that are invariant for the lifetime of a recursive port traversal
+# (index build, etc), so it is more efficient to precompute them here and pass them in
+# to child makes explicitly, instead of recomputing them tens of thousands of times.
+
.if !defined(NOPRECIOUSMAKEVARS)
.if !defined(ARCH)
ARCH!= ${UNAME} -p
.endif
-.if !defined(OSREL)
-OSREL!= ${UNAME} -r | ${SED} -e 's/[-(].*//'
-.endif
+
.if !defined(OSVERSION)
.if exists(/usr/include/sys/param.h)
OSVERSION!= ${AWK} '/^\#define[[:blank:]]__FreeBSD_version/ {print $$3}' < /usr/include/sys/param.h
@@ -79,23 +81,76 @@
OSVERSION!= ${SYSCTL} -n kern.osreldate
.endif
.endif
+
+.if !defined(_OSRELEASE)
+_OSRELEASE!= uname -r
+.endif
+.if !defined(OSREL)
+OSREL= ${_OSRELEASE:C/[-(].*//}
.endif
-INDEXDIR?= ${PORTSDIR}
-INDEXFILE?= INDEX-${OSVERSION:C/([0-9]).*/\1/}
+.if !defined(OPSYS)
+OPSYS!= ${UNAME} -s
+.endif
+
+.if ${ARCH} == "amd64" || ${ARCH} =="ia64"
+.if !defined(HAVE_COMPAT_IA32_KERN)
+HAVE_COMPAT_IA32_KERN!= if ${SYSCTL} -n compat.ia32.maxvmem >/dev/null 2>&1; then echo YES; fi
+.endif
+.endif
+.if !defined(CONFIGURE_MAX_CMD_LEN)
+CONFIGURE_MAX_CMD_LEN!= ${SYSCTL} -n kern.argmax
+.endif
+
+.if !defined(PYTHON_DEFAULT_VERSION)
+PYTHON_DEFAULT_VERSION!= make -V PYTHON_DEFAULT_VERSION USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk
+.endif
+
+.if !defined(PYTHON_DEFAULT_PORTVERSION)
+# We are caching the PYTHON_PORTVERSION of the default python version so we can reuse it in the
+# common case.
+PYTHON_DEFAULT_PORTVERSION!= make -V PYTHON_PORTVERSION USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk
+.endif
+
+.if !defined(PYTHONBASE)
+PYTHONBASE!= make -V PYTHONBASE USE_PYTHON=1 -f ${PORTSDIR}/Mk/bsd.port.mk
+.endif
+
+.if !defined(_JAVA_VERSION_LIST_REGEXP)
+_JAVA_VERSION_LIST_REGEXP!= make -V _JAVA_VERSION_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk
+.endif
+
+.if !defined(_JAVA_VENDOR_LIST_REGEXP)
+_JAVA_VENDOR_LIST_REGEXP!= make -V _JAVA_VENDOR_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk
+.endif
+
+.if !defined(_JAVA_OS_LIST_REGEXP)
+_JAVA_OS_LIST_REGEXP!= make -V _JAVA_OS_LIST_REGEXP USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk
+.endif
+
+.if !defined(_JAVA_PORTS_INSTALLED)
+_JAVA_PORTS_INSTALLED!= make -V _JAVA_PORTS_INSTALLED USE_JAVA=1 -f ${PORTSDIR}/Mk/bsd.port.mk
+.endif
+
+.if !defined(UID)
UID!= ${ID} -u
+.endif
+
.if exists(${LOCALBASE}/sbin/pkg_info)
PKG_INFO?= ${LOCALBASE}/sbin/pkg_info
.else
PKG_INFO?= /usr/sbin/pkg_info
.endif
+.if !defined(PKGINSTALLVER)
PKGINSTALLVER!= ${PKG_INFO} -P 2>/dev/null | ${SED} -e 's/.*: //'
+.endif
-.if !defined(OPSYS)
-OPSYS!= ${UNAME} -s
.endif
+INDEXDIR?= ${PORTSDIR}
+INDEXFILE?= INDEX-${OSVERSION:C/([0-9]).*/\1/}
+
# local customization of the ports tree
.if exists(${.CURDIR}/Makefile.local)
.include "${.CURDIR}/Makefile.local"
@@ -307,6 +362,8 @@
> $@
@${RM} -f $@.tmp $@.tmp2 $@.tmp3 $@.tmp4
+# Pass in the cached invariant variables to child makes.
+# XXX Why are we trying to escape these characters using regexps and not using ':Q'?
.if !defined(NOPRECIOUSMAKEVARS)
.MAKEFLAGS: \
ARCH="${ARCH:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \
@@ -314,7 +371,16 @@
OSREL="${OSREL:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \
OSVERSION="${OSVERSION:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \
UID="${UID:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \
- PKGINSTALLVER="${PKGINSTALLVER:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}"
+ PKGINSTALLVER="${PKGINSTALLVER:S/"/"'"'"/g:S/\$/\$\$/g:S/\\/\\\\/g}" \
+ HAVE_COMPAT_IA32_KERN="${HAVE_COMPAT_IA32_KERN}" \
+ CONFIGURE_MAX_CMD_LEN="${CONFIGURE_MAX_CMD_LEN}" \
+ PYTHON_DEFAULT_VERSION="${PYTHON_DEFAULT_VERSION}" \
+ PYTHON_DEFAULT_PORTVERSION="${PYTHON_DEFAULT_PORTVERSION}" \
+ PYTHONBASE="${PYTHONBASE}" \
+ _JAVA_VERSION_LIST_REGEXP="${_JAVA_VERSION_LIST_REGEXP}" \
+ _JAVA_VENDOR_LIST_REGEXP="${_JAVA_VENDOR_LIST_REGEXP}" \
+ _JAVA_OS_LIST_REGEXP="${_JAVA_OS_LIST_REGEXP}" \
+ _JAVA_PORTS_INSTALLED="${_JAVA_PORTS_INSTALLED}"
.endif
PORTSEARCH_DISPLAY_FIELDS?=name,path,info,maint,index,bdeps,rdeps,www
Index: Mk/bsd.python.mk
===================================================================
RCS file: /zoo/cvsup/FreeBSD-CVS/ports/Mk/bsd.python.mk,v
retrieving revision 1.104
diff -u -r1.104 bsd.python.mk
--- Mk/bsd.python.mk 30 May 2008 05:50:03 -0000 1.104
+++ Mk/bsd.python.mk 10 Jun 2008 12:08:49 -0000
@@ -352,14 +352,24 @@
PYTHON_VERSION?= python${_PYTHON_VERSION}
PYTHON_CMD?= ${_PYTHON_CMD}
+.if !defined(PYTHONBASE)
PYTHONBASE!= (${PYTHON_CMD} -c 'import sys; print sys.prefix' \
2> /dev/null || ${ECHO_CMD} ${LOCALBASE}) | ${TAIL} -1
+.endif
DEPENDS_ARGS+= PYTHON_VERSION=${PYTHON_VERSION}
+
+# We can only use the cached version if we are using the default python version. Otherwise it
+# should point to some other version we have installed, according to the port USE_PYTHON
+# specification
+.if !defined(PYTHON_DEFAULT_PORTVERSION) || (${PYTHON_VERSION} != ${PYTHON_DEFAULT_VERSION})
_PYTHON_PORTVERSION!= (${PYTHON_CMD} -c 'import string, sys; \
print string.split(sys.version)[0].replace("b",".b")' 2> /dev/null) | ${TAIL} -1
.if !defined(PYTHON_NO_DEPENDS) && !empty(_PYTHON_PORTVERSION)
PYTHON_PORTVERSION= ${_PYTHON_PORTVERSION}
.endif
+.elif defined(PYTHON_DEFAULT_PORTVERSION)
+PYTHON_PORTVERSION= ${PYTHON_DEFAULT_PORTVERSION}
+.endif
# Propagate the chosen python version to submakes.
.MAKEFLAGS: PYTHON_VERSION=python${_PYTHON_VERSION}
@@ -440,7 +450,9 @@
.endif
.if defined(PYEASYINSTALL_ARCHDEP)
+.if !defined(_OSRELEASE)
_OSRELEASE!= ${UNAME} -r
+.endif
PYEASYINSTALL_OSARCH?= -${OPSYS:L}-${_OSRELEASE}-${ARCH}
.endif
PYEASYINSTALL_EGG?= ${PYDISTUTILS_PKGNAME:C/[^A-Za-z0-9.]+/_/g}-${PYDISTUTILS_PKGVERSION:C/[^A-Za-z0-9.]+/_/g}-${PYTHON_VERSION:S/thon//}${PYEASYINSTALL_OSARCH}.egg
--------------040303030706050704010102--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?484EAFAC.3020208>