Date: Sun, 27 Jun 1999 01:37:04 -0700 (PDT) From: asami@freebsd.org (Satoshi - Ports Wraith - Asami) To: ports@freebsd.org Subject: speeding up INDEX generation Message-ID: <199906270837.BAA49796@silvia.hip.berkeley.edu>
next in thread | raw e-mail | index | archive | help
Hi folks,
I tried to rewrite bsd.port.subdir.mk to make the recursions
parallelizable. It is a generic change but if you have many disks, it
could help describe (i.e., INDEX generation) in particular since it is
extremely disk bound.
Here's what I came up with:
===
Index: bsd.port.mk
===================================================================
RCS file: /usr/cvs/ports/Mk/bsd.port.mk,v
retrieving revision 1.314
diff -u -r1.314 bsd.port.mk
--- bsd.port.mk 1999/06/11 11:59:10 1.314
+++ bsd.port.mk 1999/06/27 06:59:32
@@ -2295,6 +2295,9 @@
.if !target(describe)
describe:
+ @cd ${.CURDIR} && ${ECHO} `${MAKE} do-describe`
+
+do-describe:
@${ECHO} -n "${PKGNAME}|${.CURDIR}|"; \
${ECHO} -n "${PREFIX}|"; \
if [ -f ${COMMENT} ]; then \
Index: bsd.port.subdir.mk
===================================================================
RCS file: /usr/cvs/ports/Mk/bsd.port.subdir.mk,v
retrieving revision 1.30
diff -u -r1.30 bsd.port.subdir.mk
--- bsd.port.subdir.mk 1999/04/28 06:20:15 1.30
+++ bsd.port.subdir.mk 1999/06/27 07:49:50
@@ -48,30 +48,42 @@
ECHO_MSG?= echo
-_SUBDIRUSE: .USE
- @for entry in ${SUBDIR}; do \
- OK=""; \
- for dud in $$DUDS; do \
- if [ $${dud} = $${entry} ]; then \
- OK="false"; \
- ${ECHO_MSG} "===> ${DIRPRFX}$${entry} skipped"; \
- fi; \
- done; \
- if test -d ${.CURDIR}/$${entry}.${MACHINE}; then \
- edir=$${entry}.${MACHINE}; \
- elif test -d ${.CURDIR}/$${entry}; then \
- edir=$${entry}; \
- else \
+.for __target in all fetch fetch-list package package-loop extract configure \
+ build clean clean-for-cdrom clean-restricted \
+ clean-for-cdrom-list clean-restricted-list \
+ deinstall depend depends describe distclean \
+ reinstall tags checksum \
+ ignorelist
+
+.if !target(${__target})
+${__target}: ${SUBDIR:S/$/.${__target}/g}
+
+.for __subdir in ${SUBDIR}
+${__subdir}.${__target}:
+ @OK=""; \
+ for dud in $$DUDS; do \
+ if [ $${dud} = ${__subdir} ]; then \
OK="false"; \
- ${ECHO_MSG} "===> ${DIRPRFX}$${entry} non-existent"; \
+ ${ECHO_MSG} "===> ${DIRPRFX}${__subdir} skipped"; \
fi; \
- if [ "$$OK" = "" ]; then \
- ${ECHO_MSG} "===> ${DIRPRFX}$${edir}"; \
- cd ${.CURDIR}/$${edir}; \
- ${MAKE} ${.TARGET:realinstall=install} \
- DIRPRFX=${DIRPRFX}$$edir/; \
- fi; \
- done
+ done; \
+ if test -d ${.CURDIR}/${__subdir}.${MACHINE}; then \
+ edir=${__subdir}.${MACHINE}; \
+ elif test -d ${.CURDIR}/${__subdir}; then \
+ edir=${__subdir}; \
+ else \
+ OK="false"; \
+ ${ECHO_MSG} "===> ${DIRPRFX}${__subdir} non-existent"; \
+ fi; \
+ if [ "$$OK" = "" ]; then \
+ ${ECHO_MSG} "===> ${DIRPRFX}$${edir}"; \
+ cd ${.CURDIR}/$${edir}; \
+ ${MAKE} -B ${__target:realinstall=install} \
+ DIRPRFX=${DIRPRFX}$$edir/; \
+ fi
+.endfor __subdir
+.endif
+.endfor __target
${SUBDIR}::
@if test -d ${.TARGET}.${MACHINE}; then \
@@ -80,17 +92,6 @@
cd ${.CURDIR}/${.TARGET}; \
fi; \
${MAKE} all
-
-.for __target in all fetch fetch-list package package-loop extract configure \
- build clean clean-for-cdrom clean-restricted \
- clean-for-cdrom-list clean-restricted-list \
- deinstall depend depends describe distclean \
- reinstall tags checksum \
- ignorelist
-.if !target(${__target})
-${__target}: _SUBDIRUSE
-.endif
-.endfor
.if !target(install)
.if !target(beforeinstall)
===
The small bsd.port.mk patch is necessary to make sure each "describe"
line is printed as a whole, not piece by piece.
In bsd.port.subdir.mk, all the .for loops turn targets into something
like this:
describe: archivers.describe astro.describe audio.describe ...
archivers.describe:
@the real code
:
The additional "-B" in the recursion make is to cancel any -j's passed
on. Otherwise -j2 will spawn two copies in archivers and two more in
astro, etc. Other than that it's mostly just indentation changes.
With an earlier variant of this (only describe defined using a similar
.for in ${SUBDIR} but no .for for targets), I got the following on
bento (dual PII-300, 128MB main memory, ports tree on ccd of 4 SCSI
disks) for a full INDEX generation:
-j# real user sys
1 90:03 16:30 51:23
2 60:58 16:46 58:51
3 55:01 16:43 60:42
4 52:35 16:46 61:48
5 50:36 16:56 62:46
7 49:48 16:54 64:02
8 49:09 17:00 64:44
(Yes, 6 is missing. I only have five fingers to count.)
The order of INDEX lines are totally scrambled so we need to change
Tools/make_index to sort it by the second field or something. (Steve,
did we ever find out a way to sort it so that "x11/foo" comes before
"x11-toolkits/bar"?)
Comments welcome.
-PW
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199906270837.BAA49796>