From owner-freebsd-toolchain@freebsd.org Fri Nov 13 01:15:29 2015 Return-Path: Delivered-To: freebsd-toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7C5AA2CEBF for ; Fri, 13 Nov 2015 01:15:29 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 425D11E1E for ; Fri, 13 Nov 2015 01:15:29 +0000 (UTC) (envelope-from sjg@juniper.net) Received: by mailman.ysv.freebsd.org (Postfix) id 41721A2CEBE; Fri, 13 Nov 2015 01:15:29 +0000 (UTC) Delivered-To: toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28C84A2CEBD for ; Fri, 13 Nov 2015 01:15:29 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0130.outbound.protection.outlook.com [65.55.169.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D92801DFF; Fri, 13 Nov 2015 01:15:24 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from BY2PR05CA049.namprd05.prod.outlook.com (10.141.250.39) by BY1PR0501MB1381.namprd05.prod.outlook.com (10.160.107.139) with Microsoft SMTP Server (TLS) id 15.1.318.15; Thu, 12 Nov 2015 23:42:14 +0000 Received: from BN1AFFO11OLC004.protection.gbl (2a01:111:f400:7c10::161) by BY2PR05CA049.outlook.office365.com (2a01:111:e400:2c5f::39) with Microsoft SMTP Server (TLS) id 15.1.325.17 via Frontend Transport; Thu, 12 Nov 2015 23:42:14 +0000 Authentication-Results: spf=softfail (sender IP is 66.129.239.17) smtp.mailfrom=juniper.net; FreeBSD.org; dkim=none (message not signed) header.d=none;FreeBSD.org; dmarc=none action=none header.from=juniper.net; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.17 as permitted sender) Received: from p-emfe01a-sac.jnpr.net (66.129.239.17) by BN1AFFO11OLC004.mail.protection.outlook.com (10.58.53.75) with Microsoft SMTP Server (TLS) id 15.1.325.5 via Frontend Transport; Thu, 12 Nov 2015 23:42:14 +0000 Received: from magenta.juniper.net (172.17.27.123) by p-emfe01a-sac.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.123.3; Thu, 12 Nov 2015 15:42:12 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id tACNgBD83621; Thu, 12 Nov 2015 15:42:11 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [IPv6:::1]) by chaos.jnpr.net (Postfix) with ESMTP id C17C2580A9; Thu, 12 Nov 2015 15:42:10 -0800 (PST) To: Bryan Drewery CC: , , , , Subject: Re: Meta mode toolchain bootstrapping [was Re: FreeBSD targets/ out-of-date] In-Reply-To: <56450AB8.90402@FreeBSD.org> References: <55E769EF.7090908@FreeBSD.org> <4924.1441306006@chaos> <56450AB8.90402@FreeBSD.org> Comments: In-reply-to: Bryan Drewery message dated "Thu, 12 Nov 2015 13:55:04 -0800." From: "Simon J. Gerraty" X-Mailer: MH-E 8.6; nmh 1.6; GNU Emacs 24.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <8608.1447371730.1@chaos> Date: Thu, 12 Nov 2015 15:42:10 -0800 Message-ID: <13427.1447371730@chaos> X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BN1AFFO11OLC004; 1:c0o9N5io1mtHfWJ7Wr1RPO7JgHSRaavbb/cxmiFCPjRS+lyxFzSQsov1/lCPop22DVtRgcJYo4Mjki+636xVmZB6WlgcDiizOd1DQ4CvnOXiCreqFuoaHuZWzcLrpQzBImg6FzvNRzLhK9qXgVu/C02M5+iL4y4Em9TZChNoHg8vPSegdF5IlWblzZrqg+CyV8it/zy2P8K7icdjqYm+cVOs9SkqudXPwgXaTb7nEuo6kJsEW27el5B7qh/fy+iPhrpag1OjgUTRDnnxgKfkoruNUIfNj7mx4SCq6WuoDu5iHyFeJU5vGq1GHcCyPQtPxGs7KMvypisYu/PkN3fVMg== X-Forefront-Antispam-Report: CIP:66.129.239.17; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(199003)(24454002)(189002)(377454003)(52314003)(479174004)(97736004)(189998001)(97756001)(77096005)(6806005)(23726002)(19580405001)(50226001)(110136002)(81156007)(2950100001)(105596002)(19580395003)(50466002)(11100500001)(5008740100001)(450100001)(107886002)(4001430100002)(117636001)(5001960100002)(230783001)(92566002)(86362001)(50986999)(69596002)(33716001)(106466001)(76176999)(5007970100001)(87936001)(76506005)(47776003)(46406003)(57986006)(42262002)(62816006); DIR:OUT; SFP:1102; SCL:1; SRVR:BY1PR0501MB1381; H:p-emfe01a-sac.jnpr.net; FPR:; SPF:SoftFail; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0501MB1381; 2:UqDGHvqU8nUMvEDfhb2cI8kdmPyhsCcnb2WkjeQgOR4p7M1rleySnpi/Ll2RnGa88Hz3IFDmGL5ks75rOoC7Q3K7FO/8vPTVkq3Ww/cQQSa+jvZlNyZUBBYvhpA5gp/rPbAkK+cllzU2NsEcAzrTVF6BiTvPrO6Js3OoK+oMpKI=; 3:QZ5q3fWRgiecd1sMawq2AY+BYaLcf5dzUmXqDI0MG26e/5EBOlLmsISeBxg4sjvaRF9juMWOUfe8D28xyFtoIaETTdRQ0/lYfGVpqTUhPM0LS8x+uBRg/or3RelJ/0M507Ky+5j5ZmfIrQQuG6/JJMYG5t8AflLzZXkJCtPHW2dpwJPijCfGvNCBrdgsQNFQlq7F2qMKvrOuJ4hGnt/J/PcBHZ8Ruy/Emj/tuAk3ZoM=; 25:z+n1LfAZW8SHAG8vunpBjGRZ/HflRSnNbanMPGDL3cmFC3/NazMl77Mwm2HJqvpdt7FIWwIi2hU5eh5lQcSzEvDbYl+kr4WYYVw71uzS+utOVFRnAno81rhAHEVJ95cPGqDmlt4/ier9TXX7OO/QRbIH1tway6sfN4RITY4+6tlwQ29t046ZO1Q9XW5D/t13pBvGBgrXtZ/vrYL+2EJaaPHysejNN5bZUlwDc7/L3BIZcGHLU2k413hvyI8bE717dsxUC5k+2X+VzZ/VSNxu/w== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY1PR0501MB1381; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0501MB1381; 20:f+Zy19H2dlhUMkFMcGgPtbXxmwSRicQF7hEEPIp6h+jhmY/gzh8R1P22DG23W1WBs601/u056a24LLS/pPD+LDvpJqFKeHUQJUPGJqXM18zYUQdl4HxvdRR0kk/5b170dV+Uey32G9+l1pmupYqbWAlS17lHRpsZwFxrr98X38iRCP4F18WY4H44g8KC2CrzZqFrq4rlJoFgm8qFByPA5htMv86K5ei6jpMdGpTe0u3H0uSI1icugmj1oY77nl193SSyroFlwGSCj7Qa74ViIEwB7y/FosiC8HQPBNrICsk25kRBgEF3jNCXB7pK++2CxuOafJ9K8skrvwGv8ZfLmDnLMCxcjoUD1BD3XIe7zIhizG5qFYtaQnDm3XRGlBLK+2kWr+gjlcT6CIkXH75snok3u2HFFcvsXfih+TxYSX5YfMdEI3qyWWRxHOEqGc56fuTmj9IFsxxHujw3UDMNbr9z+hw4qM3vxtTRFANhF3TUxVCwHXwT93iu2sWjYEdY; 4:XUBsmXYOO3bTsNxZy1t7xP85J8iylG3F/B3bj0NKAG2XNSA5RTmQ7IQ1x8LFTostO2oCaqIVqXJHEQR2N2PsQw5cknALDzeLPv5keFwiGXyC7c5RroJULEUiOLYQYHB2p7ComK/y0m7bHOZSqURAF0joUNz5k4I7LLM5dZF3q7lUeM8rKljltiTD2D/ZmikMYTgMQ4CImUiRkulWskUVI7uxYfoq84AF4OfNL3cKjguxAqgYU/1mWDbWK5klixBWT83ybvsHNIsfKTB8kGbVJQNGjy5qAz7JfIJBf/Qx9DD3iDU6EWdJ0+cQXGTsHrR5cwalGEW9EoFBvGmUc4g20PZuh5kzU3/L/55PrqJXqlLHQQFvOQBzgHoFUslN9Bna X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001)(10201501046); SRVR:BY1PR0501MB1381; BCL:0; PCL:0; RULEID:; SRVR:BY1PR0501MB1381; X-Forefront-PRVS: 07584EDBCD X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY1PR0501MB1381; 23:j/I0gXFHgwysss/qhszlE3eNRUfbpf5KzT/iuee?= =?us-ascii?Q?2quXYx2+8W/S4EFKx4QP2bzQBCZK6LMtb5nwT+ilkXaMLv+tEkfxp1Y0FzM6?= =?us-ascii?Q?fjGWZtkxjPbjSXcKdP88xQVACPakep36V4oSVxwQVSLE7OWG0UnrYRPooylE?= =?us-ascii?Q?As+G4C9+3Mm9GHkpkFF3c4fNXmretyx3JlBqCtSmfVdy1DsK8LGrKAaAL8uB?= =?us-ascii?Q?iQkzWWDXf9+P61JEY4xEzdBz0weFSfzxjkxblK7qZRqsiEiCIaOQmogehV27?= =?us-ascii?Q?Oh4YWaKt5/6crBP29TSoNEcyA1GesRjuDwyhS2UOJPbsI5K0youx3UJLzNUa?= =?us-ascii?Q?c7TwbCw8ORipaCmolFoVmX08AoIeZ+aU5nvKCb8IwSfIeZ9MtZYW3FkfcdfC?= =?us-ascii?Q?vCB7ueq76kl0FLh1+VoRjC412Y/Z00Dm0BP1wpi4F7ucgExy1PpwXPfU1Tbl?= =?us-ascii?Q?+y/gjHHeO0tEXA4PMvFM7hg1PYYPjT1eEE6yK3ky10xHHOYd5zL1zCL8yje8?= =?us-ascii?Q?mTYNsPBrQfQ8z/tbFZag4kuQ3zFnDXTS6CaK+3Gv/vGFbqeHP72zS9g9CXn9?= =?us-ascii?Q?gre3fkHGiE8cwJyc7IqZs56C5ur83L7Reu9DYwOfsKHjgePEnjt8SYOftVXc?= =?us-ascii?Q?8PNBGdVqQOw2A1wZDK5ZDY/nFaA5b8exwXjria4aROCtzaspy1baZgrgYfgA?= =?us-ascii?Q?wgfUjvdRyy96BTTFjPNB6J4+xib5ZFk0xL/WsRi9SqqEN8/pM4tkqupwLom7?= =?us-ascii?Q?jzIX43xh5q0VM5jyqJU/tM4r2t4IyczIXCNi1aBQ+YZw+tih4C5dM27KJxVi?= =?us-ascii?Q?QAp8GSc74x2MIb+DLgrln7vz+/3xKKytyplz9D6sM459LH6lCWDCUpeteNgb?= =?us-ascii?Q?SRsJp6V7rTLfQHijNDDtI+HIjk7WMlgkVr/5XM2GQuaHwN3HYifl4gFmvXo5?= =?us-ascii?Q?MAzfn4mThgCgll7kNGP10IWUMVyz+VftHdJvAqdVlVrxGWoiPBLJAiLpfuA/?= =?us-ascii?Q?xxsz44GXmJRNfQMaXUuLY7TB9iUr0GP0WX0qF7SmuXTYSHv4N5qWnn0kEVk1?= =?us-ascii?Q?1aZYB6tF8QP3chVjBwMMe42HT2QziIlKNM2GxbSyLJTI9sTCHMBzmubbBCPT?= =?us-ascii?Q?1JmALAzrcrTsLgf3XfPfCuBFckgsTvca76CVucK/u5ixkNxOdateuvRv4/Pw?= =?us-ascii?Q?ewJz/CqUWwJliMWisif+ZpXL9++naqhXX0gtC?= X-Microsoft-Exchange-Diagnostics: 1; BY1PR0501MB1381; 5:clyM358Y99Y60Gn5KgETa/FggnYw0Cm4mwRnTzVFqENIaiXn8KaJOUpGcb7R5pZbspQUE34Bqn1ujjGZC8x9JwAbt3ljbaDq3IqDw8m4y3NLSreFfVdbwzb9s2L+B2QxivoYRHTweGeAqLPZoXZ/Mg==; 24:Gc+zlk47i3j8QWAgxgmJM+oTYvqBeVZNQcyVDMhktkB0FVQTbUxURw7H3Ha9V9ugC0P797/0EUWmaTaf+ZVXGIEQl7a/5Io34tQSjQxSln4=; 20:3u9qFcimvYmL0I3VMrtwfEWRtc1Khgi9hNT2OguYnsq9pnQoY35bL/MsGNSRtTB7Lnlcl9xEN6vYO/Msc7S8bA== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Nov 2015 23:42:14.2347 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.17]; Helo=[p-emfe01a-sac.jnpr.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0501MB1381 X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Nov 2015 01:15:29 -0000 Bryan Drewery wrote: > On 9/3/2015 11:46 AM, Simon J. Gerraty wrote: > > Anyway, bootstrap-toolchain leverages src/Makefile.inc1 to build an > > initial set of tools. It then attempts to use that to build toolchain > > for MACHINE=host which is currently failing because in-tree libelf and > > libdwarf are needed and libelf needs sys/sys/elf_common.h but but that > > doesn't currently get staged for host (we try to minimize what we put in > > host stage tree). > > Using the special hack you came up with for lib/libelf, and a lot of > other fixes for replacing binutils with elftoolchain, I've managed to > get the targets/pseduo/bootstrap-tools build working again. I will > commit this likely today or tomorrow. Cool - thanks > However... > > It may be better to simply skip targets/pseudo/toolchain.host > > that will work so long as we set TOOLSDIR to point to the stuff built by > > Makefile.inc1 > > > > Depending on where we are with external toolchian support that would > > make sense - might be able to skip bootstrap-toolchain completely > > which would be nice. > > I think this is more right because there's many more people thinking > about, testing daily, and maintaining the bootstrap logic in > Makefile.inc1. The way targets/pseudo/bootstrap-tools works currently > lead to bitrot and breakage with the elftoolchain replacement. It will > very easily happen again. I'd be favor of removing the need completely. All that is required is *some* means of producing a viable toolchain in such a way that you can point a variable at it. > What I plan to do is make pseudo/targets/bootstrap-tools always build > all of cross-tools,etc, from Makefile.inc1 respecting its own internal > logic for when to bootstrap the compiler (without us telling it to skip > the bootstrap compiler) and then add pseudo/targets/bootstrap-tools as a > global DIRDEP. Given the use of cookies, this will only be done once per > meta build, but it at least won't be a manual step anymore. For the That can still add a lot of time to build, and IIRC building clang alone poses problems for smaller machines (though using a mutex for linking clang bits could at least serialize the huge linker steps so as to avoid exhausting memory). I quite like the way NetBSD allow separating the tools from the rest of the build. I do the equivalent of 'make tools' very rarely - and so do not care how [in]efficient it is. IIRC it will throw an error if your tools are incompatible - prompting an update. I would suggest if you want to be able to hook bootstrap-tools in always, that you do it via a knob so it can be disabled. > record, I do plan to extend .MAKE.MODE=meta to buildworld and its > bootstrapping as well, which will give a lot of incremental build > improvements to it. WITH_META_FILES should give you improvements already in that regard. > This means that the meta build will then default to bootstrapping the > compiler, which we really must do to build the source tree reliably. > There's really no reason the meta build should default to not > bootstrapping the compiler. Setting WITHOUT_CROSS_COMPILER or > WITHOUT_CLANG_BOOTSTRAP will avoid this and use just the host compiler > as the meta build does now. So there's no real problem for those people > who want to skip the clang bootstrap. Ok, so long as there is a way to do so. Otherwise we'd need to add it locally for our own builds - where we need to use the toolchains provided by our compiler team. > Then, I will work on the project of "building clang less" which will > avoid building the bootstrap compiler for both the meta mode build and > the buildworld build (and universe eventually) if /usr/bin/cc satisfies > the needs (can cross compile the TARGET and is "new enough" compared to > the version we want). This is not trivial but I think it is possible and > it will make everyone happy! Indeed. As I say, NetBSD have this reasonably sorted. But of course they have 2k line shell script driving a lot of it ;-) > Related tidbit, using WITHOUT_CROSS_COMPILER (or WITHOUT_*_BOOTSTRAP) > with buildworld leads to a broken build currently since it is the user > basically asking to use their default external toolchain of /usr/bin/*, > but the logic does not kick in to use --sysroot against WORLDTMP. I plan > to fix this soon as well. Assuming that you have previously built the correct toolchain it should be valid to have something like -DWITH_TOOLSDIR -DWITHOUT_BOOTSTRAP_TOOLSDIR (or -DWITHOUT_TOOLSDIR_BOOTSTRAP) Such that the build logic is identical - all that is being skipped is the expense of re-generating the toolchain