From owner-freebsd-ports@freebsd.org Mon May 11 20:40:40 2020 Return-Path: Delivered-To: freebsd-ports@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7E8A72EB48F for ; Mon, 11 May 2020 20:40:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-21.consmr.mail.gq1.yahoo.com (sonic305-21.consmr.mail.gq1.yahoo.com [98.137.64.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49LXrP3QGXz46Mn for ; Mon, 11 May 2020 20:40:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: HGVJGoUVM1mrghAwREIpvZDwqjYZf.ypla6tQMDVHByiK3.iMMiNVAv_3we9tBU 8W1xezozDJcPMOuR1rfnty60pjpEGT4b_B.4kA0iS0vFPkLMVhxpNEbdMt2oWE_PovSxJCInqIr2 565A_RfvFi5kytvtueMdFKAigH57Sp9Mc0zisyj4gZ2haCclqysbcsT2Ngt0qflAVT_hw7ocwoTm .oHJQP_spkG1wc7wCa1UTabOLk_JdpIPYNceF8jDE9.nCuO0iSiDl94e0zlt3Xtrv9K4i71EB0U_ R_HzA2nwZZj2TgfpAjnGvXcPPM.CQROyhWKSyXKOJgQvR0ISCTy2iKf.tLXUBa7sb5QOMlfJsSBD limV37fm93cVE5YDJHzU1Sismw_PtyrAQ3xLjYnVO33ckjvB.GnqoN1jEA4IbRmyJvGNurP.W7IP IHqDudgUWShvl81rHETRoTQbP.OBPeKMWRHjje53p4t1qmQMLNN_L1WalGhbx0qNPdBD_qpXmwff VqF7thULeBrYnLR02GoOtBzgcqiaN6iCf0wMFRTbSlW_wrlw4nollrTfUkE4sd.9pND25s183cJs rR7HERxmkKvS05VgqucQGXAKRHD9brQ.EL1uIGioHJHKzIgzdK8b6qf8IjUbwRkd3p2EpYP4Wg7o XX66aDEL_u0tGacpSkO1lXpEsdjWvo0WCQ.ZfG_Cl8sD5QBo29BvdGASz4lISalDxkovle4eJsXE XEpD7UUDZZmOxtU637KWsybtRCwaPgYpL0ThgGaYD5oP3RoeEHKVrj.jLZsDp9M2qc3iEP6w7DEu dpm3IpXOigHrp8xLeTqSkyl5ngLDeI3Y0RnH2Z34wKH4kQXol8UPoJ7YUqMS7VtUc02bPLv4bEwV IE5yqcxiFjMAh.SncSvsnb0C7sCP63MJoFz2LShCxSH7QM_XBrLXmBWbR1zuBXq537G5XMucYK.U .AOa4wI2IgrN0neJ9Z9_bjbYTe5WNn4K43EyrJ_TB.Ww0_d4MO0c3HVBjDVBRUVkAb.gsqVihPCw AGgdm3EnqVz1EK.3qpEErtYNkwCWGEMlKCiuqLY5eEfmTW8x7.bIS2AgFhemRt.OFciyYxD3Mduc hNbKCqSPMWzwIC99gzx0TP7t38gB0XDMlU8XWQ2VmL4TrrxXxntwZq7c0GWLPyEAr83kgfuzpIwC hiSOvuyAgogbxutbosgEa1hVwd.8JnJBuIgTlI4jH3cphUSpS49.PCxTZOJsuAYpaQaYjAh8wwzF l_fufvGBLJiQuHg0b28TH9NR9XQ.pU0y5V468FkzdW6ooCYIC168HO.MPGWoAdiahcANmKR2JkIk sTPzpgLpqNqBeMRJDgn22Jq.rr30BnOgtjVKPlHDsFjS1oL99XC9jBbZckr.W.1uDH0sTCqUYiwn 3zD_NT_b_H8WfvLiWSLL5Bg-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Mon, 11 May 2020 20:40:34 +0000 Received: by smtp426.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 5a92c327b34723dc6de4200878519818; Mon, 11 May 2020 20:40:29 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: Poudriere - Compile ALL Ports From: Mark Millard In-Reply-To: Date: Mon, 11 May 2020 13:40:27 -0700 Cc: "info@netocean.de" , FreeBSD ports Content-Transfer-Encoding: quoted-printable Message-Id: <2611B3DD-43C3-420D-A909-A8C51EB7EC1E@yahoo.com> References: <625BBD86-9740-41D0-8199-CF1B3ACB5320.ref@yahoo.com> <625BBD86-9740-41D0-8199-CF1B3ACB5320@yahoo.com> To: "Patrick M. Hausen" X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49LXrP3QGXz46Mn X-Spamd-Bar: - X-Spamd-Result: default: False [-1.94 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.60)[-0.600,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.84)[-0.842,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (3.48), ipnet: 98.137.64.0/21(0.83), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[84.64.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[84.64.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 May 2020 20:40:41 -0000 On 2020-May-11, at 11:34, Patrick M. Hausen wrote: > I=E2=80=99m confused by this thread. isn=E2=80=99t the point of = Poudriere > bulk builds to When I listed: # poudriere jail -l JAILNAME VERSION ARCH METHOD TIMESTAMP = PATH FBSDFSSDjail 13.0-CURRENT amd64 null 2018-11-09 = 08:58:53 /usr/obj/DESTDIRs/clang-amd64-installworld-poud I probably should have explicitly pointed out the use of "METHOD" "null" as the type of context I was covering. Poudiere has many options. One of them for "poudriere jail" is: -m method Specifies which method to use to create the jail. = (default:http) . . . null This option can be used to import an = existing directory that already contains an = installed system. The path must be specified = with -M path. It is expected that this = directory be installed to with the following: /usr/src# make installworld = DESTDIR=3DPATH DB_FROM_SRC=3D1 /usr/src# make distrib-dirs = DESTDIR=3DPATH DB_FROM_SRC=3D1 /usr/src# make distribution = DESTDIR=3DPATH DB_FROM_SRC=3D1 The path will be null-mounted during = builds. It will not be copied at the time of = running jail. Deleting the jail will attempt = to re- vert any files changed by poudriere. . . . This is one way of testing development contexts that might not exist in svn yet (or ever). There is also src=3DPATH and tar=3DPATH and the like. Part of my guess was that such a "not from scratch" context might be in use. The guess could be wrong for all I know. The rest of the below is tied the the "-m null" sort of jail context: It is not covering all poudriere jail methods. > - start with a clean jail If one manually makes the jail unclean by doing chroot and things like installing ports, poudriere does not clean out what one did: it takes the jail (mostly) as-is and uses it to make the reference jail for the run. (That in turn is used to make the jails for jobs.) So it is the poudriere user's responsibility to have the jail's content clean before starting poudriere. (Remember: -m null as a poudriere jail context is the context being covered.) If started clean, poudriere will keep it clean. If started dirty, poudriere will keep it dirty. > - build the first port in the list and all dependencies The order is dependencies have their packages built first and dependencies are later used by being temporarily installed in the job's jail before building what was dependent. After the dependent package is packaged, the temporary material is eliminated before the next port is built in that jail. > - store the packages for later incorporation into the pkg repo Previously built packages are also used to install what a dependent port's build depends on, even in the same overall bulk run. > - clean out the jail completely There are multiple jails involved, especially for parallel jobs. poudriere makes a reference jail off the original user jail (off of /usr/obj/DESTDIRs/clang-amd64-installworld-poud in my example). It then makes jails for the parallel jobs off the reference jail as I understand. These are what get poudriere's equivalent of what a "clean out" would be (between jobs that reuse the specific temporary jail). For the most part poudriere leaves the likes of /usr/obj/DESTDIRs/clang-amd64-installworld-poud alone and works with its temporary reference and job jails. > - start with the next port in the list With parallel jobs and complicated dependencies, the concept of "next" is itself somewhat complicated but respects the dependency structure by not starting anything too early for what all it depends on to be available to be temporarily installed in job's jail. > - =E2=80=A6 >=20 > How can there ever be a build conflict unless there is a bug > like port A depends on B and ImageMagick 7, while B is > lagging somewhat behind and depends on ImageMagick 6? If one has poudriere create the build's reference jail from a user jail that already has ports installed, then the reference and job jails will also already have those ports installed. That in turn can lead to conflicts. > But if ImageMagick 6 and 7 are not dependencies but intended > target packages, doesn=E2=80=99t Poudriere build each of them in a = shiny > new empty environment? The new environment is only empty if the original user jail was already empty. (Again: the -m null poudriere jail context is the intended coverage of the statements.) > At least that=E2=80=99s what I thought our Poudriere was doing all day > and night and we never had a build conflict like this. Bugs like > the aforementioned aside. I do not think that the original report listed what options were in use for the poudriere jail involved, such as -m method. One way to show some of that is (using my example): # poudriere jail -jFBSDFSSDjail -i Jail name: FBSDFSSDjail Jail version: 13.0-CURRENT Jail arch: amd64 Jail method: null Jail mount: /usr/obj/DESTDIRs/clang-amd64-installworld-poud Jail fs: =20 Jail updated: 2018-11-09 08:58:53 I do a similar thing with the ports tree used: # poudriere ports -l PORTSTREE METHOD TIMESTAMP PATH default null 2017-08-14 21:07:05 /usr/ports Again a "null" METHOD where I'm supplying my own materials. Again it allows for testing development materials that are not checked into svn. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)