From owner-freebsd-current@FreeBSD.ORG Fri Nov 12 13:05:55 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1CC5516A4CE; Fri, 12 Nov 2004 13:05:55 +0000 (GMT) Received: from tigra.ip.net.ua (tigra.ip.net.ua [82.193.96.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id CAA9143D48; Fri, 12 Nov 2004 13:05:52 +0000 (GMT) (envelope-from ru@ip.net.ua) Received: from localhost (rocky.ip.net.ua [82.193.96.2]) by tigra.ip.net.ua (8.12.11/8.12.11) with ESMTP id iACD5p7E022292; Fri, 12 Nov 2004 15:05:51 +0200 (EET) (envelope-from ru@ip.net.ua) Received: from tigra.ip.net.ua ([82.193.96.10]) by localhost (rocky.ipnet [82.193.96.2]) (amavisd-new, port 10024) with LMTP id 59426-10; Fri, 12 Nov 2004 15:05:49 +0200 (EET) Received: from heffalump.ip.net.ua (heffalump.ip.net.ua [82.193.96.213]) by tigra.ip.net.ua (8.12.11/8.12.11) with ESMTP id iACD5nwI022289 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 12 Nov 2004 15:05:49 +0200 (EET) (envelope-from ru@ip.net.ua) Received: (from ru@localhost) by heffalump.ip.net.ua (8.13.1/8.13.1) id iACD5nS3086315; Fri, 12 Nov 2004 15:05:49 +0200 (EET) (envelope-from ru) Date: Fri, 12 Nov 2004 15:05:48 +0200 From: Ruslan Ermilov To: Poul-Henning Kamp Message-ID: <20041112130548.GA86162@ip.net.ua> References: <20041112115824.GA85834@ip.net.ua> <97256.1100261516@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ibTvN161/egqYuK8" Content-Disposition: inline In-Reply-To: <97256.1100261516@critter.freebsd.dk> User-Agent: Mutt/1.5.6i X-Virus-Scanned: by amavisd-new at ip.net.ua cc: Harti Brandt cc: current@freebsd.org Subject: Re: [TEST] make -j patch [take 2] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Nov 2004 13:05:55 -0000 --ibTvN161/egqYuK8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 12, 2004 at 01:11:56PM +0100, Poul-Henning Kamp wrote: > I have explained this several times already: You should just leave > the submakes on their own and let them balance as they see fit. >=20 > I spent a lot of time on parallel/clustered make for a customer and > the outcome of that is very clear: Leave the stuff alone after you > have said your overall intention. >=20 > Obviously you are not going to trust me on this, since you seem to > have very strong ideas on how you think it works in practice, even > though you don't seem to have any actual data to back it up with. >=20 > I suggest that you actually try out your ideas in _practice_, not > just with a few "proof of concept" makefiles made for the purpuse, > but try to actually _run_ it on real world jobs for some weeks while > you carefully gater statistics and examine the interactions between > the parameters given, where given, resulting time to build and > loading of the machine(s). >=20 > At the very least, do not commit your patch until you have managed > to come up with at least one instance of real world data where it > is a good idea. >=20 I gave you one example already, and Harti gave you another. "make universe" builds 6 worlds today. By starting it with "make -j12", you build 6 worlds in parallel, and another 6 jobs are made available for building these worlds. When one world finishes, 7 jobs will be available for building 5 other worlds, etc. This is fine as long as you're interested in the final result of "make universe". When testing a commit candidate change with "make universe", I want to detect the breakage as early as possible. This can be achieved by building less worlds in parallel, and giving more parallelism to individual world builds. The overall time of "make universe" won't change significantly. If you're by chance familiar with how you can ask Oracle to optimize your queries, you can either optimize for ALL ROWS or for FIRST ROWS, and it's up to you to decide on this. The "rows" in "make universe" are results of doing buildworld and buildkernel, and you have control over this. Do you disagree with anything of the above? But since resetting MAKE_JOBS_FIFO before starting a sub-make will have a similar effect, I won't be committing my patch to make(1). Can you agree to the below patch? It doesn't change anything except when JFLAG is set explicitly to request another level of parallelism for building individual worlds and kernels? %%% Index: Makefile =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /home/ncvs/src/Makefile,v retrieving revision 1.310 diff -u -r1.310 Makefile --- Makefile 6 Nov 2004 03:14:26 -0000 1.310 +++ Makefile 12 Nov 2004 12:58:33 -0000 @@ -252,6 +252,11 @@ # with a reasonable chance of success, regardless of how old your # existing system is. # +.if defined(JFLAG) && !empty(JFLAG) +UMAKE=3D unset MAKE_JOBS_FIFO && ${MAKE} ${JFLAG} +.else +UMAKE=3D ${MAKE} +.endif universe: universe_prologue universe_prologue: @echo "--------------------------------------------------------------" @@ -264,7 +269,7 @@ .ORDER: universe_prologue universe_${mach} universe_epilogue universe_${mach}: @echo ">> ${mach} started on `LC_ALL=3DC date`" - -cd ${.CURDIR} && ${MAKE} ${JFLAG} buildworld \ + -cd ${.CURDIR} && ${UMAKE} buildworld \ TARGET_ARCH=3D${arch} TARGET=3D${mach} \ __MAKE_CONF=3D/dev/null \ > _.${mach}.buildworld 2>&1 @@ -290,7 +295,7 @@ find [A-Z]*[A-Z] -type f -maxdepth 0 ! -name NOTES buildkernels: .for kernel in ${KERNCONFS} - -cd ${.CURDIR} && ${MAKE} ${JFLAG} buildkernel \ + -cd ${.CURDIR} && ${UMAKE} buildkernel \ KERNCONF=3D${kernel} \ __MAKE_CONF=3D/dev/null \ > _.${TARGET}.${kernel} 2>&1 %%% Cheers, --=20 Ruslan Ermilov ru@FreeBSD.org FreeBSD committer --ibTvN161/egqYuK8 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQFBlLUsqRfpzJluFF4RArdhAJ40ZpNl7L06up6w92U3DeqQR9f82gCdE6d9 yyVx33c2xyhjILpULtUYiz4= =2fT1 -----END PGP SIGNATURE----- --ibTvN161/egqYuK8--