From owner-svn-ports-head@freebsd.org Wed Nov 9 12:14:48 2016 Return-Path: Delivered-To: svn-ports-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A96DC37DC9; Wed, 9 Nov 2016 12:14:48 +0000 (UTC) (envelope-from mat@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 079A324A; Wed, 9 Nov 2016 12:14:47 +0000 (UTC) (envelope-from mat@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uA9CElNh057413; Wed, 9 Nov 2016 12:14:47 GMT (envelope-from mat@FreeBSD.org) Received: (from mat@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uA9CEl2d057412; Wed, 9 Nov 2016 12:14:47 GMT (envelope-from mat@FreeBSD.org) Message-Id: <201611091214.uA9CEl2d057412@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mat set sender to mat@FreeBSD.org using -f From: Mathieu Arnold Date: Wed, 9 Nov 2016 12:14:47 +0000 (UTC) To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r425793 - head/Mk X-SVN-Group: ports-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-ports-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the ports tree for head List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Nov 2016 12:14:48 -0000 Author: mat Date: Wed Nov 9 12:14:47 2016 New Revision: 425793 URL: https://svnweb.freebsd.org/changeset/ports/425793 Log: Optimize COPYTREE_* macros. The main problem slowing down those macros was that it was ending up calling chmod(1) once for each file and directory. It was doing this because it was running find(1) in the source tree and needed to chmod in the target tree, so it had to append the full path to the chmod(1) calls, which made it impossible to use -exec +. Rewrite the -exec calls to call sh and cd into it before running chmod(1) and do so on as many files as we can using the -exec + construct. While there, optimize a bit more after figuring out that it is possible to only use one find(1) to run two -exec for different options. Also switch to using the more powerfull modern regular expressions instead of the usual basic regular expressions by using find -E. This will allow the use of | which as no basic regular expressions equivalent (as in, for example, "*.(html|txt)") as well as the + and ? keywords, and back references. About the speed gain, I did a few tests, using time's builtin from csh (table with all the data available in the review), comparing the home baked versions of COPYTREE_SHARE found in ports, it boils down to: | version | www/sit (~2k) | www/mathjax (~30k) | +------------+------------------+--------------------+ | home baked | 0.032u 0.104s | 0.946u 7.446s | | current | 0.281u 0.649s | 5.446u 18.626s | | new | 0.024u 0.081s | 0.770u 7.012s | PR: 213614 Submitted by: mat Exp-run by: antoine Sponsored by: Absolight Differential Revision: https://reviews.freebsd.org/D8283 Modified: head/Mk/bsd.port.mk (contents, props changed) Modified: head/Mk/bsd.port.mk ============================================================================== --- head/Mk/bsd.port.mk Wed Nov 9 11:19:20 2016 (r425792) +++ head/Mk/bsd.port.mk Wed Nov 9 12:14:47 2016 (r425793) @@ -2057,14 +2057,15 @@ MAKE_ENV+= ${INSTALL_MACROS} SCRIPTS_ENV+= ${INSTALL_MACROS} # Macro for copying entire directory tree with correct permissions -COPYTREE_BIN= ${SH} -c '(${FIND} -d $$0 $$2 | ${CPIO} -dumpl $$1 >/dev/null \ - 2>&1) && \ - ${FIND} -d $$0 $$2 -type d -exec chmod 755 $$1/{} \; && \ - ${FIND} -d $$0 $$2 -type f -exec chmod ${BINMODE} $$1/{} \;' -- -COPYTREE_SHARE= ${SH} -c '(${FIND} -d $$0 $$2 | ${CPIO} -dumpl $$1 >/dev/null \ - 2>&1) && \ - ${FIND} -d $$0 $$2 -type d -exec chmod 755 $$1/{} \; && \ - ${FIND} -d $$0 $$2 -type f -exec chmod ${SHAREMODE} $$1/{} \;' -- +# In the -exec shell commands, we add add a . as the first argument, it would +# end up being $0 aka the script name, which is not part of $@, so we force it +# to be able to use $@ directly. +COPYTREE_BIN= ${SH} -c '(${FIND} -Ed $$0 $$2 | ${CPIO} -dumpl $$1 >/dev/null 2>&1) && \ + ${FIND} -Ed $$0 $$2 \( -type d -exec ${SH} -c '\''cd '\''$$1'\'' && chmod 755 "$$@"'\'' -- . {} + \ + -o -type f -exec ${SH} -c '\''cd '\''$$1'\'' && chmod ${BINMODE} "$$@"'\'' -- . {} + \)' -- +COPYTREE_SHARE= ${SH} -c '(${FIND} -Ed $$0 $$2 | ${CPIO} -dumpl $$1 >/dev/null 2>&1) && \ + ${FIND} -Ed $$0 $$2 \( -type d -exec ${SH} -c '\''cd '\''$$1'\'' && chmod 755 "$$@"'\'' -- . {} + \ + -o -type f -exec ${SH} -c '\''cd '\''$$1'\'' && chmod ${SHAREMODE} "$$@"'\'' -- . {} + \)' -- # The user can override the NO_PACKAGE by specifying this from # the make command line