From owner-freebsd-ports Sun Jun 27 1:37:46 1999 Delivered-To: freebsd-ports@freebsd.org Received: from dfw-ix10.ix.netcom.com (dfw-ix10.ix.netcom.com [206.214.98.10]) by hub.freebsd.org (Postfix) with ESMTP id 053F214C96 for ; Sun, 27 Jun 1999 01:37:43 -0700 (PDT) (envelope-from asami@cs.berkeley.edu) Received: (from smap@localhost) by dfw-ix10.ix.netcom.com (8.8.4/8.8.4) id DAA13979 for ; Sun, 27 Jun 1999 03:37:41 -0500 (CDT) Received: from sji-ca44-84.ix.netcom.com(209.111.212.212) by dfw-ix10.ix.netcom.com via smap (V1.3) id rma013967; Sun Jun 27 03:37:08 1999 Received: (from asami@localhost) by silvia.hip.berkeley.edu (8.9.3/8.6.9) id BAA49796; Sun, 27 Jun 1999 01:37:04 -0700 (PDT) Date: Sun, 27 Jun 1999 01:37:04 -0700 (PDT) Message-Id: <199906270837.BAA49796@silvia.hip.berkeley.edu> X-Authentication-Warning: silvia.hip.berkeley.edu: asami set sender to asami@cs.berkeley.edu using -f To: ports@freebsd.org Subject: speeding up INDEX generation From: asami@freebsd.org (Satoshi - Ports Wraith - Asami) Sender: owner-freebsd-ports@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi folks, I tried to rewrite bsd.port.subdir.mk to make the recursions parallelizable. It is a generic change but if you have many disks, it could help describe (i.e., INDEX generation) in particular since it is extremely disk bound. Here's what I came up with: === Index: bsd.port.mk =================================================================== RCS file: /usr/cvs/ports/Mk/bsd.port.mk,v retrieving revision 1.314 diff -u -r1.314 bsd.port.mk --- bsd.port.mk 1999/06/11 11:59:10 1.314 +++ bsd.port.mk 1999/06/27 06:59:32 @@ -2295,6 +2295,9 @@ .if !target(describe) describe: + @cd ${.CURDIR} && ${ECHO} `${MAKE} do-describe` + +do-describe: @${ECHO} -n "${PKGNAME}|${.CURDIR}|"; \ ${ECHO} -n "${PREFIX}|"; \ if [ -f ${COMMENT} ]; then \ Index: bsd.port.subdir.mk =================================================================== RCS file: /usr/cvs/ports/Mk/bsd.port.subdir.mk,v retrieving revision 1.30 diff -u -r1.30 bsd.port.subdir.mk --- bsd.port.subdir.mk 1999/04/28 06:20:15 1.30 +++ bsd.port.subdir.mk 1999/06/27 07:49:50 @@ -48,30 +48,42 @@ ECHO_MSG?= echo -_SUBDIRUSE: .USE - @for entry in ${SUBDIR}; do \ - OK=""; \ - for dud in $$DUDS; do \ - if [ $${dud} = $${entry} ]; then \ - OK="false"; \ - ${ECHO_MSG} "===> ${DIRPRFX}$${entry} skipped"; \ - fi; \ - done; \ - if test -d ${.CURDIR}/$${entry}.${MACHINE}; then \ - edir=$${entry}.${MACHINE}; \ - elif test -d ${.CURDIR}/$${entry}; then \ - edir=$${entry}; \ - else \ +.for __target in all fetch fetch-list package package-loop extract configure \ + build clean clean-for-cdrom clean-restricted \ + clean-for-cdrom-list clean-restricted-list \ + deinstall depend depends describe distclean \ + reinstall tags checksum \ + ignorelist + +.if !target(${__target}) +${__target}: ${SUBDIR:S/$/.${__target}/g} + +.for __subdir in ${SUBDIR} +${__subdir}.${__target}: + @OK=""; \ + for dud in $$DUDS; do \ + if [ $${dud} = ${__subdir} ]; then \ OK="false"; \ - ${ECHO_MSG} "===> ${DIRPRFX}$${entry} non-existent"; \ + ${ECHO_MSG} "===> ${DIRPRFX}${__subdir} skipped"; \ fi; \ - if [ "$$OK" = "" ]; then \ - ${ECHO_MSG} "===> ${DIRPRFX}$${edir}"; \ - cd ${.CURDIR}/$${edir}; \ - ${MAKE} ${.TARGET:realinstall=install} \ - DIRPRFX=${DIRPRFX}$$edir/; \ - fi; \ - done + done; \ + if test -d ${.CURDIR}/${__subdir}.${MACHINE}; then \ + edir=${__subdir}.${MACHINE}; \ + elif test -d ${.CURDIR}/${__subdir}; then \ + edir=${__subdir}; \ + else \ + OK="false"; \ + ${ECHO_MSG} "===> ${DIRPRFX}${__subdir} non-existent"; \ + fi; \ + if [ "$$OK" = "" ]; then \ + ${ECHO_MSG} "===> ${DIRPRFX}$${edir}"; \ + cd ${.CURDIR}/$${edir}; \ + ${MAKE} -B ${__target:realinstall=install} \ + DIRPRFX=${DIRPRFX}$$edir/; \ + fi +.endfor __subdir +.endif +.endfor __target ${SUBDIR}:: @if test -d ${.TARGET}.${MACHINE}; then \ @@ -80,17 +92,6 @@ cd ${.CURDIR}/${.TARGET}; \ fi; \ ${MAKE} all - -.for __target in all fetch fetch-list package package-loop extract configure \ - build clean clean-for-cdrom clean-restricted \ - clean-for-cdrom-list clean-restricted-list \ - deinstall depend depends describe distclean \ - reinstall tags checksum \ - ignorelist -.if !target(${__target}) -${__target}: _SUBDIRUSE -.endif -.endfor .if !target(install) .if !target(beforeinstall) === The small bsd.port.mk patch is necessary to make sure each "describe" line is printed as a whole, not piece by piece. In bsd.port.subdir.mk, all the .for loops turn targets into something like this: describe: archivers.describe astro.describe audio.describe ... archivers.describe: @the real code : The additional "-B" in the recursion make is to cancel any -j's passed on. Otherwise -j2 will spawn two copies in archivers and two more in astro, etc. Other than that it's mostly just indentation changes. With an earlier variant of this (only describe defined using a similar .for in ${SUBDIR} but no .for for targets), I got the following on bento (dual PII-300, 128MB main memory, ports tree on ccd of 4 SCSI disks) for a full INDEX generation: -j# real user sys 1 90:03 16:30 51:23 2 60:58 16:46 58:51 3 55:01 16:43 60:42 4 52:35 16:46 61:48 5 50:36 16:56 62:46 7 49:48 16:54 64:02 8 49:09 17:00 64:44 (Yes, 6 is missing. I only have five fingers to count.) The order of INDEX lines are totally scrambled so we need to change Tools/make_index to sort it by the second field or something. (Steve, did we ever find out a way to sort it so that "x11/foo" comes before "x11-toolkits/bar"?) Comments welcome. -PW To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message