From owner-dev-commits-src-main@freebsd.org Thu Dec 31 17:17:51 2020 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A16644C8CC6; Thu, 31 Dec 2020 17:17:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D6FGR48ZWz3Kp4; Thu, 31 Dec 2020 17:17:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 817AD14510; Thu, 31 Dec 2020 17:17:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 0BVHHpxK092949; Thu, 31 Dec 2020 17:17:51 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 0BVHHp4G092948; Thu, 31 Dec 2020 17:17:51 GMT (envelope-from git) Date: Thu, 31 Dec 2020 17:17:51 GMT Message-Id: <202012311717.0BVHHp4G092948@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Kyle Evans Subject: git: ac5f382a9d0a - main - stand: properly declare subdir deps or .WAIT, do parallel build MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kevans X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: ac5f382a9d0a26685b92b49abb845d3b30ea5f91 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Commit messages for the main branch of the src repository." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Dec 2020 17:17:51 -0000 The branch main has been updated by kevans: URL: https://cgit.FreeBSD.org/src/commit/?id=ac5f382a9d0a26685b92b49abb845d3b30ea5f91 commit ac5f382a9d0a26685b92b49abb845d3b30ea5f91 Author: Kyle Evans AuthorDate: 2020-12-31 17:15:45 +0000 Commit: Kyle Evans CommitDate: 2020-12-31 17:15:45 +0000 stand: properly declare subdir deps or .WAIT, do parallel build buildworld already runs the stand build in parallel[1], so make it easier to identify ordering issues by properly establishing dependencies or adding .WAIT where needed. Everything in stand/ relies on libsa, either directly or indirectly, because libsa build is where the stand headers get installed and it gets linked in most places. Interpreters depend on their libs, machine dirs usually depend on top-level libs that are getting built and at least one of the interpreter flavors. For i386, order btx/libi386/libfirewire before everything else using a big-ol-.WAIT hammer. btx is the most common dependency, but the others are used sporadically. This seems to be where the race reporting on the mailing list is- AFAICT, the following sequence is happening: 1.) One of the loaders gets built based on stale btx/btxldr 2.) btx/btxldr gets rebuilt 3.) installworld triggers loader rebuild because btx was rebuilt after This seems like the most plausible explanation, as they've verified system time and timestamps. While we're here, let's switch stand/ over to a completely parallel build so we can work out these kinds of issues in isolation rather than in the middle of a larger build. Reviewed by: bdragon, sjg, tsoome Tested by: bdragon (-j1024, no failures, significant speed improvement) Differential Revision: https://reviews.freebsd.org/D23411 --- stand/Makefile | 40 ++++++++++++++++++++++++++++++++++++++-- stand/Makefile.amd64 | 4 ++++ stand/Makefile.inc | 2 ++ stand/efi/Makefile | 6 +++++- stand/i386/Makefile | 13 ++++++++++--- 5 files changed, 59 insertions(+), 6 deletions(-) diff --git a/stand/Makefile b/stand/Makefile index bacf39e26eca..d7bcb4bfbca2 100644 --- a/stand/Makefile +++ b/stand/Makefile @@ -11,22 +11,44 @@ LIB32LIST=libsa ficl liblua S.yes+= libsa +S.${MK_LOADER_OFW}+= libofw +S.${MK_FDT}+= fdt + S.${MK_FORTH}+= ficl S.${MK_FORTH}+= forth S.${MK_LOADER_LUA}+= liblua S.${MK_LOADER_LUA}+= lua -S.${MK_FDT}+= fdt -S.${MK_LOADER_OFW}+= libofw S.yes+= defaults S.yes+= man +.if ${MK_FORTH} != "no" +INTERP_DEPENDS+= forth +.endif +.if ${MK_LOADER_LUA} != "no" +INTERP_DEPENDS+= lua +.endif + .include S.${MK_EFI}+= efi S.${MK_LOADER_UBOOT}+= uboot +.if defined(LIB32LIST) +LIB32DEPENDS= ${LIB32LIST:S/$/32/} +.endif + .if exists(${.CURDIR}/${MACHINE}/.) S.yes+= ${MACHINE} +SUBDIR_DEPEND_${MACHINE}+= ${INTERP_DEPENDS} +.if ${MK_FDT} != "no" +SUBDIR_DEPEND_${MACHINE}+= fdt +.endif +.if ${MK_LOADER_UBOOT} != "no" +SUBDIR_DEPEND_${MACHINE}+= uboot +.endif +.if ${MK_LOADER_OFW} != "no" +SUBDIR_DEPEND_${MACHINE}+= libofw +.endif .endif # Build the actual subdir list from S.yes, adding in the 32-bit @@ -36,6 +58,20 @@ SUBDIR+=${_x} .if defined(LIB32LIST) && ${LIB32LIST:M${_x}} SUBDIR+=${_x}32 .endif +.if ${_x} != "libsa" +SUBDIR_DEPEND_${_x}+= libsa +SUBDIR_DEPEND_${_x}32+= libsa32 +.endif .endfor +# Remaining dependencies +SUBDIR_DEPEND_libsa32+= libsa + +SUBDIR_DEPEND_forth+= ficl +SUBDIR_DEPEND_lua+= liblua + +SUBDIR_DEPEND_efi+= fdt + +SUBDIR_PARALLEL= yes + .include diff --git a/stand/Makefile.amd64 b/stand/Makefile.amd64 index b2b918ebed5b..9ee3649071b2 100644 --- a/stand/Makefile.amd64 +++ b/stand/Makefile.amd64 @@ -2,3 +2,7 @@ S.yes+= userboot S.yes+= i386 + +SUBDIR_DEPEND_userboot+= ${INTERP_DEPENDS} +# These won't get tacked on in an amd64 build +SUBDIR_DEPEND_i386+= ${LIB32DEPENDS} ${INTERP_DEPENDS} diff --git a/stand/Makefile.inc b/stand/Makefile.inc index d7b9d228287e..aa7cfa14542f 100644 --- a/stand/Makefile.inc +++ b/stand/Makefile.inc @@ -1,3 +1,5 @@ # $FreeBSD$ +SUBDIR_PARALLEL= yes + .include "defs.mk" diff --git a/stand/efi/Makefile b/stand/efi/Makefile index f6cbede52bd5..94cb5bba5638 100644 --- a/stand/efi/Makefile +++ b/stand/efi/Makefile @@ -4,8 +4,12 @@ NO_OBJ=t .include +SUBDIR.yes+= libefi SUBDIR.${MK_FDT}+= fdt -SUBDIR.yes+= libefi boot1 gptboot +SUBDIR.yes+= .WAIT + +SUBDIR.yes+= boot1 gptboot + SUBDIR.${MK_FORTH}+= loader_4th SUBDIR.${MK_LOADER_LUA}+= loader_lua SUBDIR.yes+= loader_simp diff --git a/stand/i386/Makefile b/stand/i386/Makefile index a9d402acf609..9aa33462f309 100644 --- a/stand/i386/Makefile +++ b/stand/i386/Makefile @@ -4,10 +4,15 @@ NO_OBJ=t .include -SUBDIR.yes= mbr pmbr boot0 boot0sio btx boot2 cdboot gptboot \ - isoboot libi386 - +# Almost everything else here relies on btxldr, so we must make sure it's built +# before everything else proceeds so we don't end up building against a stale +# btxldr and ending up with a build-during-install scenario. +SUBDIR.yes+= btx libi386 SUBDIR.${MK_LOADER_FIREWIRE}+= libfirewire +SUBDIR.yes+= .WAIT + +SUBDIR.yes+= mbr pmbr boot0 boot0sio boot2 cdboot gptboot \ + isoboot SUBDIR.${MK_FORTH}+= loader_4th SUBDIR.${MK_LOADER_LUA}+= loader_lua @@ -18,4 +23,6 @@ SUBDIR.yes+= pxeldr SUBDIR.${MK_LOADER_ZFS}+= zfsboot gptzfsboot +SUBDIR_DEPEND_pxeldr+= loader_${LOADER_DEFAULT_INTERP} + .include