From owner-freebsd-toolchain@freebsd.org Fri Nov 13 01:38:27 2015 Return-Path: Delivered-To: freebsd-toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C883A2D4EE for ; Fri, 13 Nov 2015 01:38:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3CAD81C53 for ; Fri, 13 Nov 2015 01:38:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 39B1AA2D4EC; Fri, 13 Nov 2015 01:38:27 +0000 (UTC) Delivered-To: toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 393F0A2D4EA for ; Fri, 13 Nov 2015 01:38:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 1FB7F1C52; Fri, 13 Nov 2015 01:38:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 118E91136; Fri, 13 Nov 2015 01:38:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id BFC9D162ED; Fri, 13 Nov 2015 01:38:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id rmZXlsD2M3Yt; Fri, 13 Nov 2015 01:38:23 +0000 (UTC) Subject: Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date] DKIM-Filter: OpenDKIM Filter v2.9.2 mail.xzibition.com 2FA73162E8 To: "Simon J. Gerraty" References: <55E769EF.7090908@FreeBSD.org> <4924.1441306006@chaos> <56450AB8.90402@FreeBSD.org> <13427.1447371730@chaos> Cc: brooks@FreeBSD.org, imp@FreeBSD.org, emaste@FreeBSD.org, toolchain@FreeBSD.org From: Bryan Drewery Openpgp: id=F9173CB2C3AAEA7A5C8A1F0935D771BB6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Organization: FreeBSD Message-ID: <56453F0D.90206@FreeBSD.org> Date: Thu, 12 Nov 2015 17:38:21 -0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <13427.1447371730@chaos> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0oqmDci2hpNGxNBp5Rtq0eMQ7TI1r758V" X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Nov 2015 01:38:27 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --0oqmDci2hpNGxNBp5Rtq0eMQ7TI1r758V Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 11/12/2015 3:42 PM, Simon J. Gerraty wrote: > Bryan Drewery wrote: >=20 >> On 9/3/2015 11:46 AM, Simon J. Gerraty wrote: >>> Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an >>> initial set of tools. It then attempts to use that to build toolchai= n >>> for MACHINE=3Dhost which is currently failing because in-tree libelf = and >>> libdwarf are needed and libelf needs sys/sys/elf_common.h but but tha= t >>> doesn't currently get staged for host (we try to minimize what we put= in >>> host stage tree). >> >> Using the special hack you came up with for lib/libelf, and a lot of >> other fixes for replacing binutils with elftoolchain, I've managed to >> get the targets/pseduo/bootstrap-tools build working again. I will >> commit this likely today or tomorrow. >=20 > Cool - thanks > =20 >> However... >=20 >>> It may be better to simply skip targets/pseudo/toolchain.host >>> that will work so long as we set TOOLSDIR to point to the stuff built= by >>> Makefile.inc1=20 >>> >>> Depending on where we are with external toolchian support that would >>> make sense - might be able to skip bootstrap-toolchain completely >>> which would be nice. >> >> I think this is more right because there's many more people thinking >> about, testing daily, and maintaining the bootstrap logic in >> Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently >> lead to bitrot and breakage with the elftoolchain replacement. It will= >> very easily happen again. >=20 > I'd be favor of removing the need completely. >=20 > All that is required is *some* means of producing a viable toolchain in= > such a way that you can point a variable at it. >=20 >> What I plan to do is make pseudo/targets/bootstrap-tools always build >> all of cross-tools,etc, from Makefile.inc1 respecting its own internal= >> logic for when to bootstrap the compiler (without us telling it to ski= p >> the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as= a >> global DIRDEP. Given the use of cookies, this will only be done once p= er >> meta build, but it at least won't be a manual step anymore. For the >=20 > That can still add a lot of time to build, and IIRC building clang alon= e > poses problems for smaller machines (though using a mutex for linking > clang bits could at least serialize the huge linker steps so as to avoi= d > exhausting memory). >=20 > I quite like the way NetBSD allow separating the tools from the rest of= > the build. I do the equivalent of 'make tools' very rarely - and so > do not care how [in]efficient it is. > IIRC it will throw an error if your tools are incompatible - prompting > an update. >=20 > I would suggest if you want to be able to hook bootstrap-tools in > always, that you do it via a knob so it can be disabled. I would just add some knob like WITH_BOOTSTRAP_TOOLCHAIN or WITH_EXTERNAL_TOOLCHAIN to make this and the clang bootstrap just be skipped and resort to the way the meta build works now. >=20 >> record, I do plan to extend .MAKE.MODE=3Dmeta to buildworld and its >> bootstrapping as well, which will give a lot of incremental build >> improvements to it. >=20 > WITH_META_FILES should give you improvements already in that regard. Yes, it's a step. We'll need cookies in a lot of places too. I wish WITH_META_MODE had been WITH_META_BUILD or WITH_DIRDEPS_BUILD so I could check for "META_MODE" in the buildworld world and for discussion sake. It seems I can use ${.MAKE.MODE:M*meta*} but that :U is needed in all the uses. I'm not sure yet if :U really is needed. We have some ${MK_META_MODE} checks now around some cookies that would need to change for what I'm planning. >=20 >> This means that the meta build will then default to bootstrapping the >> compiler, which we really must do to build the source tree reliably. >=20 >> There's really no reason the meta build should default to not >> bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or >> WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler= >> as the meta build does now. So there's no real problem for those peopl= e >> who want to skip the clang bootstrap. >=20 > Ok, so long as there is a way to do so. > Otherwise we'd need to add it locally for our own builds - where we nee= d > to use the toolchains provided by our compiler team. Yes for sure. As noted above. At Isilon when a developer is building they have no interest in building the toolchain 99% of the time. This is why I am so committed to making this automatically skip the toolchain if possible. As for having an external compiler from a team, that's kind of already in the "external compiler" realm, so a WITH_EXTERNAL_COMPILER would make sense to short-circuit all of what I'm wanting. Then you can add it to your local make.conf or local.sys.mk and not have any change in behavior. >=20 >> Then, I will work on the project of "building clang less" which will >> avoid building the bootstrap compiler for both the meta mode build and= >> the buildworld build (and universe eventually) if /usr/bin/cc satisfie= s >> the needs (can cross compile the TARGET and is "new enough" compared t= o >> the version we want). This is not trivial but I think it is possible a= nd >> it will make everyone happy! >=20 > Indeed. As I say, NetBSD have this reasonably sorted. > But of course they have 2k line shell script driving a lot of it ;-) Yes the NetBSD build, behavior wise, really impresses me. >=20 >> Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP) >> with buildworld leads to a broken build currently since it is the user= >> basically asking to use their default external toolchain of /usr/bin/*= , >> but the logic does not kick in to use --sysroot against WORLDTMP. I pl= an >> to fix this soon as well. >=20 > Assuming that you have previously built the correct toolchain > it should be valid to have something like -DWITH_TOOLSDIR > -DWITHOUT_BOOTSTRAP_TOOLSDIR (or -DWITHOUT_TOOLSDIR_BOOTSTRAP) > Such that the build logic is identical - all that is being skipped is > the expense of re-generating the toolchain >=20 --=20 Regards, Bryan Drewery --0oqmDci2hpNGxNBp5Rtq0eMQ7TI1r758V Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJWRT8OAAoJEDXXcbtuRpfP/xwH/A8moFzEflsJJvpXjEWxnCy+ jH0vXJsQTzAfLZv7LeqvN8GnPbI/hta/fwhXnLJXdePYWdmriHkQhYtk1IIccRz6 8Da/li4i+8Acn/Pf+P2Ka0Wlzf+upeNYlURiKt+p1oZqVkzlBQp32kpnPiy/r+ID DXuAAg2fRbND13VWsptx07sCzVOtMw0eAaGu9Gbgyf4K+u9bJMMJ6sJa5KoPbZKl df6zNhnwZsWY67kt4cL2JgW+whrCX+Mn8cVNLKahHgUNbPxL3qUTIJPbefu4JB35 jhp9hDz+2dU/JIOULTMK9ULaAH2+rjTn9K2a8I8xSL64YypIg+KdzRqrkY2BHt8= =P1il -----END PGP SIGNATURE----- --0oqmDci2hpNGxNBp5Rtq0eMQ7TI1r758V--