Date: Tue, 16 Dec 2014 13:00:21 +0800 From: Jia-Shiun Li <jiashiun@gmail.com> To: Ian Lepore <ian@freebsd.org> Cc: freebsd-arch <freebsd-arch@freebsd.org> Subject: _bootstrap-tools parallel build (was: Re: CFR, CFT: Fine-grained SUBDIR dependencies for parallel builds Message-ID: <CAHNYxxO0F%2BHojpq2As--PeDn0WP6AVew5%2B-4c24bG7TC4hzsrQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
On Sat, May 31, 2014 at 10:04 PM, Ian Lepore <ian@freebsd.org> wrote:
>
> The parallelism in the bootstrap stuff in Makefile.inc1 is done with a
> different-but-similar mechanism, and it would be nice to fix that to
> just use bsd.subdir.mk instead of almost-duplicating it inline. That's
> on a longer-term to-do list for me.
>
Hi Ian & all,
I did it manually for _bootstrap-tools anyway. Patches attached.
It reduces 'make -j24 _botostrap-tools' time from ~55 sec. to ~25 sec.
on a 12C IVB-EP.
It scales more linearly in comparison to single-job ~250 sec on the
same machine.
Main blockers were tblgen & groff.
On -j4 it reduces time from ~82 sec. to ~60 sec. too.
It passed 'make -j24 universe' except the seemingly broken i386 kernels.
Any comments?
-Jia-Shiun
[-- Attachment #2 --]
Index: Makefile.inc1
===================================================================
--- Makefile.inc1 (revision 275699)
+++ Makefile.inc1 (working copy)
@@ -1313,11 +1313,7 @@
usr.bin/compile_et
.endif
-# Please document (add comment) why something is in 'bootstrap-tools'.
-# Try to bound the building of the bootstrap-tool to just the
-# FreeBSD versions that need the tool built at this stage of the build.
-bootstrap-tools: .MAKE
-.for _tool in \
+BT_DIRS= \
${_clang_tblgen} \
${_kerberos5_bootstrap_tools} \
${_dtrace_tools} \
@@ -1340,15 +1336,40 @@
usr.sbin/config \
${_crunch} \
${_nmtree} \
- ${_vtfontcvt}
- ${_+_}@${ECHODIR} "===> ${_tool} (obj,depend,all,install)"; \
- cd ${.CURDIR}/${_tool} && \
- ${MAKE} DIRPRFX=${_tool}/ obj && \
- ${MAKE} DIRPRFX=${_tool}/ depend && \
- ${MAKE} DIRPRFX=${_tool}/ all && \
- ${MAKE} DIRPRFX=${_tool}/ DESTDIR=${MAKEOBJDIRPREFIX}/legacy install
+ ${_vtfontcvt} \
+
+.for X in ${BT_DIRS}
+# Generate rules
+${X}: .PHONY
+ ${_+_}@${ECHODIR} "===> ${X} (obj,depend,all,install)"; \
+ cd ${.CURDIR}/${X} && \
+ ${MAKE} DIRPRFX=${X}/ obj && \
+ ${MAKE} DIRPRFX=${X}/ depend && \
+ ${MAKE} DIRPRFX=${X}/ all && \
+ ${MAKE} DIRPRFX=${X}/ DESTDIR=${MAKEOBJDIRPREFIX}/legacy install
.endfor
+# ${_clang_tblgen} dependencies
+usr.bin/clang/tblgen: lib/clang/libllvmtablegen lib/clang/libllvmsupport
+usr.bin/clang/clang-tblgen: lib/clang/libllvmtablegen lib/clang/libllvmsupport
+
+# ${_kerberos5_bootstrap_tools} dependencies
+kerberos5/tools/slc: kerberos5/lib/libroken
+kerberos5/tools/asn1_compile: kerberos5/lib/libroken
+
+# below dependencies not tested but replicated as-is
+# ${_yacc}
+usr.bin/yacc: lib/liby
+# ${_m4}
+usr.bin/m4: lib/libohash
+# ${_nmtree}
+usr.sbin/nmtree: lib/libnetbsd
+
+# Please document (add comment) why something is in 'bootstrap-tools'.
+# Try to bound the building of the bootstrap-tool to just the
+# FreeBSD versions that need the tool built at this stage of the build.
+bootstrap-tools: ${BT_DIRS}
+
#
# build-tools: Build special purpose build tools
#
Index: gnu/usr.bin/groff/Makefile
===================================================================
--- gnu/usr.bin/groff/Makefile (revision 275699)
+++ gnu/usr.bin/groff/Makefile (working copy)
@@ -2,4 +2,6 @@
SUBDIR= contrib doc font man src tmac
+SUBDIR_PARALLEL=
+
.include <bsd.subdir.mk>
Index: gnu/usr.bin/groff/src/Makefile
===================================================================
--- gnu/usr.bin/groff/src/Makefile (revision 275699)
+++ gnu/usr.bin/groff/src/Makefile (working copy)
@@ -2,4 +2,10 @@
SUBDIR= libs devices preproc roff utils
+SUBDIR_PARALLEL=
+SUBDIR_DEPEND_devices= libs
+SUBDIR_DEPEND_preproc= libs
+SUBDIR_DEPEND_roff= libs
+SUBDIR_DEPEND_utils= libs
+
.include <bsd.subdir.mk>
Index: gnu/usr.bin/groff/src/devices/Makefile
===================================================================
--- gnu/usr.bin/groff/src/devices/Makefile (revision 275699)
+++ gnu/usr.bin/groff/src/devices/Makefile (working copy)
@@ -2,4 +2,6 @@
SUBDIR= grodvi grohtml grolbp grolj4 grops grotty
+SUBDIR_PARALLEL=
+
.include <bsd.subdir.mk>
Index: gnu/usr.bin/groff/src/preproc/Makefile
===================================================================
--- gnu/usr.bin/groff/src/preproc/Makefile (revision 275699)
+++ gnu/usr.bin/groff/src/preproc/Makefile (working copy)
@@ -2,4 +2,6 @@
SUBDIR= eqn grn html pic refer soelim tbl
+SUBDIR_PARALLEL=
+
.include <bsd.subdir.mk>
Index: gnu/usr.bin/groff/src/roff/Makefile
===================================================================
--- gnu/usr.bin/groff/src/roff/Makefile (revision 275699)
+++ gnu/usr.bin/groff/src/roff/Makefile (working copy)
@@ -2,4 +2,6 @@
SUBDIR= groff grog nroff psroff troff
+SUBDIR_PARALLEL=
+
.include <bsd.subdir.mk>
Index: gnu/usr.bin/groff/src/utils/Makefile
===================================================================
--- gnu/usr.bin/groff/src/utils/Makefile (revision 275699)
+++ gnu/usr.bin/groff/src/utils/Makefile (working copy)
@@ -2,4 +2,6 @@
SUBDIR= addftinfo afmtodit hpftodit indxbib lkbib lookbib pfbtops tfmtodit
+SUBDIR_PARALLEL=
+
.include <bsd.subdir.mk>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHNYxxO0F%2BHojpq2As--PeDn0WP6AVew5%2B-4c24bG7TC4hzsrQ>