From owner-svn-src-head@FreeBSD.ORG Thu Apr 24 17:17:40 2014 Return-Path: Delivered-To: svn-src-head@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 52B85C17; Thu, 24 Apr 2014 17:17:39 +0000 (UTC) Received: from aslan.scsiguy.com (aslan.scsiguy.com [70.89.174.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D21AE15CA; Thu, 24 Apr 2014 17:17:38 +0000 (UTC) Received: from benscottlt.sldomain.com (207-225-98-3.dia.static.qwest.net [207.225.98.3]) (authenticated bits=0) by aslan.scsiguy.com (8.14.8/8.14.8) with ESMTP id s3OHHRBf024552 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 24 Apr 2014 11:17:29 -0600 (MDT) (envelope-from gibbs@scsiguy.com) Content-Type: text/plain; charset=windows-1253 Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: svn commit: r263778 - in head: bin lib lib/clang sbin share/mk usr.bin usr.sbin From: "Justin T. Gibbs" In-Reply-To: Date: Thu, 24 Apr 2014 11:17:22 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <201403262230.s2QMUdH6021943@svn.freebsd.org> <20140327181245.GA69977@stack.nl> <7A86F5E9-DBE9-4D3F-B166-C02F8386B722@FreeBSD.org> <1398086131.1124.371.camel@revolution.hippie.lan> To: Warner Losh X-Mailer: Apple Mail (2.1874) Cc: src-committers@FreeBSD.org, Ian Lepore , Jilles Tjoelker , svn-src-all@FreeBSD.org, Dimitry Andric , svn-src-head@FreeBSD.org X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Apr 2014 17:17:40 -0000 Why wouldn=92t we fix these with explicit subdir dependencies (e.g: = subdira: subdirb)? I=92m pretty sure I had this working with a hacked = up bsd.subdir.mk at a previous job, but my memory is hazy. =97 Justin =20 On Apr 21, 2014, at 7:35 AM, Warner Losh wrote: > (sorry for the top post) >=20 > This looks good to my eye. I=92d be tempted to toss in a comment about = the > __wait=3D.WAIT construct is due to the primitive nature of bmake=92s = parser > so it runs afoul of the .for/.if construction rules and this is needed = to expand > the for variable. It tripped me up when I looked at it, until I = recalled a comment > from similar code in NetBSD. >=20 > Warner >=20 > On Apr 21, 2014, at 7:15 AM, Ian Lepore wrote: >=20 >> On Thu, 2014-03-27 at 20:44 +0100, Dimitry Andric wrote: >>> On 27 Mar 2014, at 19:12, Jilles Tjoelker wrote: >>>> On Thu, Mar 27, 2014 at 11:05:00AM -0600, Warner Losh wrote: >>>>> On Mar 26, 2014, at 4:30 PM, Dimitry Andric = wrote: >>>>>> Author: dim >>>>>> Date: Wed Mar 26 22:30:38 2014 >>>>>> New Revision: 263778 >>>>>> URL: http://svnweb.freebsd.org/changeset/base/263778 >>>>=20 >>>>>> Log: >>>>>> Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to = process >>>>>> all the SUBDIR entries in parallel, instead of serially. Apply = this >>>>>> option to a selected number of Makefiles, which can greatly speed = up the >>>>>> build on multi-core machines, when using make -j. >>>>=20 >>>>>> This can be extended to more Makefiles later on, whenever they = are >>>>>> verified to work correctly with parallel building. >>>>=20 >>>>> Why not have this =91opt out=92 rather than =91opt in=92 like it = is now? Are >>>>> there any known bad dependencies this introduces? >>>>=20 >>>> I'm paranoid about build systems ;) It is easy to add dependencies >>>> across directories and as long as directories are built in = sequence, >>>> nothing goes wrong. >>>>=20 >>>> In fact, I had enabled SUBDIR_PARALLEL in sys/modules/Makefile as = well, >>>> but this caused mysterious failures with some kernels such as mips >>>> ADM5120. >>>=20 >>> There are a bunch of other parts that don't really like parallel = builds >>> at the moment. For example, gnu/usr.bin/binutils needs its = libraries >>> (libbfd.a, etc) built first, before it can link the programs. = Similar >>> for gnu/usr.bin/cc, which needs libiberty, libcpp, etc before being = able >>> to build the rest of gcc. >>>=20 >>> Most of these cases can hopefully be solved by adding .WAIT targets = at >>> strategic points in the SUBDIR lists, but this also needs a bit of = extra >>> logic in bsd.subdir.mk. >>>=20 >>> -Dimitry >>>=20 >>=20 >> It turns out I needed the .WAIT functionality to use SUBDIR_PARALLEL = for >> $work, so I came up with the attached, does this look okay to commit? >>=20 >> -- Ian >>=20 >> diff -r 67802e319fc6 share/mk/bsd.subdir.mk >> --- a/share/mk/bsd.subdir.mk Sun Apr 20 21:01:07 2014 -0600 >> +++ b/share/mk/bsd.subdir.mk Mon Apr 21 06:59:37 2014 -0600 >> @@ -4,10 +4,10 @@ >> # The include file contains the default targets >> # for building subdirectories. >> # >> -# For all of the directories listed in the variable SUBDIRS, the >> +# For all of the directories listed in the variable SUBDIR, the >> # specified directory will be visited and the target made. There is >> # also a default target which allows the command "make subdir" where >> -# subdir is any directory listed in the variable SUBDIRS. >> +# subdir is any directory listed in the variable SUBDIR. >> # >> # >> # +++ variables +++ >> @@ -42,7 +42,7 @@ distribute: >>=20 >> _SUBDIR: .USE >> .if defined(SUBDIR) && !empty(SUBDIR) && !defined(NO_SUBDIR) >> - @${_+_}for entry in ${SUBDIR}; do \ >> + @${_+_}for entry in ${SUBDIR:N.WAIT}; do \ >> if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \ >> ${ECHODIR} "=3D=3D=3D> = ${DIRPRFX}$${entry}.${MACHINE_ARCH} (${.TARGET:realinstall=3Dinstall})"; = \ >> edir=3D$${entry}.${MACHINE_ARCH}; \ >> @@ -57,7 +57,7 @@ distribute: >> done >> .endif >>=20 >> -${SUBDIR}: .PHONY >> +${SUBDIR:N.WAIT}: .PHONY >> ${_+_}@if test -d ${.TARGET}.${MACHINE_ARCH}; then \ >> cd ${.CURDIR}/${.TARGET}.${MACHINE_ARCH}; \ >> else \ >> @@ -65,13 +65,18 @@ distribute: >> fi; \ >> ${MAKE} all >>=20 >> +__wait=3D.WAIT >> .for __target in all all-man checkdpadd clean cleandepend cleandir \ >> depend distribute lint maninstall manlint \ >> obj objlink realinstall regress tags \ >> ${SUBDIR_TARGETS} >> .ifdef SUBDIR_PARALLEL >> +__subdir_targets=3D >> .for __dir in ${SUBDIR} >> -${__target}: ${__target}_subdir_${__dir} >> +.if ${__wait} =3D=3D ${__dir} >> +__subdir_targets+=3D .WAIT >> +.else >> +__subdir_targets+=3D ${__target}_subdir_${__dir} >> ${__target}_subdir_${__dir}: .MAKE >> @${_+_}set -e; \ >> if test -d ${.CURDIR}/${__dir}.${MACHINE_ARCH}; then \ >> @@ -85,7 +90,9 @@ distribute: >> fi; \ >> ${MAKE} ${__target:realinstall=3Dinstall} \ >> DIRPRFX=3D${DIRPRFX}$$edir/ >> +.endif >> .endfor >> +${__target}: ${__subdir_targets} >> .else >> ${__target}: _SUBDIR >> .endif >=20 >=20 >=20