From nobody Sat Jul 10 20:49:21 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 22A1E127A820 for ; Sat, 10 Jul 2021 20:49:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-22.consmr.mail.gq1.yahoo.com (sonic302-22.consmr.mail.gq1.yahoo.com [98.137.68.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GMhwS5CZVz4vRM for ; Sat, 10 Jul 2021 20:49:28 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625950167; bh=jV2mM9XiCl23Jqpb6r5yz+R+n0Xo4zyjyjrwYTnVIW4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Y3Oi3x9UJokSUN6T92mVNNVNoBFnyxFT2dfOeP2uc4TloPH73p/v1At00r04Z6Klnk1RSYrIJBprLCUScxFmQ6QFKDFChRuSUqJGBwCaBr4jHkf6YrvK2XkL7RQCjKVUXJhuNKgVcb99Krgb9LiMGKFr0WqIsMnKfYCKGCDcaCOQ1X8htE64Ad0LM3oEDUIUs9oixUdhe2QNrRmlI4ph9CRmtH8v+B38fJFBekrNNXcCNG3OzpHJZwu1HqcWUR0hfXd3T3kaAJbh7XAcABH1M/a1uYY+BBnr9P/+w7FKj10z8F86pec5xJb52JYAZcjAsCNlbFUi8TNjafdJxHyeew== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625950167; bh=ltgAdtUa7V2U8JzMa0QBO/8mTDgm//dTqYnQtnqcIMP=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=QZvr3TvtKK8s/AafCnkMOSKvq3BON3U8N6ggSQckF4szLS65fu3C1vYKLUcchQ821cO84Ap7YuTJRiEui13ktQIzn7DDZT9xgZjGubf193XPboWstf0/Jiiqw7yFaRphxGLYJtAzXqRaxqYrrwsRG/vK7gfSQ258Ky1XYnfVC5tTnMwzJEVr3z4fIbJ4K6mCKd2VGqa3qvNuMqVvcVo4MXO5hGQkkx2eBH6ypIKg6JD6LS5cjDwC5QDH96SN2SM57eI4+A38uKmYJS2t7DfZdCaiANRkFcwFSF4gLyZ/1H39XTj88TLmAeckvPhFYchBhHzDpNOwojg33/pvXzT/Nw== X-YMail-OSG: v6CpMJUVM1kvxeKDX5xdDXQVkk3ejY7D9xvfytU.3_4IKv_5_MvLYQIiwgi5l8_ a7PlRKd_43tYk78_fS1vvfXiVkaXq8K4CWvqHIU.igAYhgxopRyLSpCMMJyNev6sr4YTiCIrXL6X 9aUsm3AgOM3KddduIgQFEo2IBDjwd6zfWLIMw8lShXb3W2EZZVOnmjbQxs4mN6DITh7Yny6Wa8SA b9UZwoAn6I0utnBbHzD8fABo.N1hNDxfwRGwqYzT6T9fBhJ07kaeM74eLYCx.LkwkG3zOKvuCbVa .J0rwM9H99FqPZZCBoJTtsP0LxaYjQKjRzGtXKVGhHw8UPi9wiU6AEUFHhBbQvQ4FvfGMvaqAfIC wg96ZNqRKDTdy4nyUbBdIcZ3AUwagU8VZaKm4ZSJlG6GYmJGK797CksAzlvzoH6CNID3uWiHG4iI tLS69XWw2Ht7ubV8WvAW0TIif4FjPK7h0sGgedQX_we772CYR7vwKJZUvvQXsIYCdLPgiibGAoHM OnxRAXol20lQfKa3udeTvbOYIDee34ROoepPwxWlqPcFwfYdk8.izVZyqiUJteZlMSrKVsB9QQIo OrfR1RysnZiIijfp1IYNyw.vHY.Ijib8HMMNX3Hd45t4aJ0jONNyfhFnmAwboipA5ls1EGilR9s0 qSVCM5MZL0udtCoVXPcjf2YMBX6iyde_XWKtCw7wrh3hd.I0q7cULb7md2WrtpozKq4uPBtXJRBb KI5Qpn1sF4SsVLJkiJqcS43gyPfdn_S_reXAYQ8.Birk1bGA7PE6H2JhbscV8f3n6YSf7E9SvXB4 K9.MgGbsGyJNuOSkMUVMECZxZ3ur3h5q3OepSYOHvRLmxEgfhRwgxZ8_tf7p6M.6QcMUDI5tQ9bp Hph_qhsrfYnRfUbvlnc_6OA1_vuUoX7WUMrpfhbDbCd_u4B6KO5HfSL8xsucy0uDpszalQqYoarb YA_Y.HpIM_nPsw10Z4Lh07cYEDbDvEUK2IWFygZlbLws87OAT9j5kpfOeKFG1nun5IJSJaorcFPH ukIvOsWNN3fEVT8.k1ranLiRA6crNsj9lxiknAiXKI8STPGG8JEcXhyNkLNpYuJq6tefiKzu8jrY gvas9298wrNkSoZRBbYHHEwf0SY4pmLqq0oJU5V8rNP75MNq.j1BVjrIx_l4cXlvQwi66FtUA2zd aHBcX0N8EfHoapf0gq6PWefkIvjP3VDtt.74TTFhXTrt.WTNJnw68DGRJYU2UKOhzGoyX_ehpRdG EzsGe4RGjYa69ecDJvV.TP.X1wIRUKp6s5QDplypUyQ8hOsl3.A_JX0kxY3.rnRkd0JdGJ33BqYB IW4sR_dasdxk14_zPIDojIfIo_zwXLq8UIEXeTy.bmV2Su4_NQZyU_T57tkma__mh7BnTT4iiiAf _TDlQ.X8fJ9qRl77zS6yth.mSylf6JVWPqA4puy4jPIQjF2x0SLL8K9dNWMXXM3rCazCWPj977Tj owWiIjCF7jRsO05i7tLsTrTf869PdubyH8eu0reC2sb_4l75gq7TENgNqr4Y8S0cNgRQ0fvhgxdf RjZzJIfe9Rjvo.3WzqFpLKiZs2RFNLdVTQO0fv6dpcGZF0D_2upPDqpZg8BhqsRiNGKKNbG0_gyA E9vGt0_UOSpy1Y6X_GTA1SySbvM4aKjx_wSM7F4Su45mkyteTY_v2Pg2xEIf4_30zu6Wv_ZDis2m 1ySXZTWnbbJFp43VtVRlDW5BU89Zmex.NAKHAu.hf80usqm17pYQuY.iZmJqbsezy1IdVwBAdLmK jYNFBupNIGkX.sxo6_dgGYiWDledFhOU0UBzVCw9s0LDtlii9AFpIgkJvpM1Aq902USgdvKiFfYH AF9Hevwwno.L_4UM4zjsiu897vQGpg3soVEWT2pynswzJzStluglXzLeJ0OuVTe2Y12x1wIFMZlj pO0NKGVOhTm4jDWeuPUmaunTCv_qPp7nFqG4XkHsWh7Pfvs2zkeIB04aCBqhMs_pZWQIR4QJPTrT xu0hYvhZx2bkErjGtHIooXn6MnPDmauoyy85JzYDGWwHrc95tKPoQrntcbVm.JhtyOShc8bhFrMC yRnhmZa0F_g_YxMbnCaNc79dpKs9kIG7npMwmyO38xJIehink6xi6CKz8aHhivtdd25ZcoXqM_Zb PNy3.pGcOoCuiRwdh4hIYgiFJKwlX9eYXgnAYiriuaskxHhNojUzx4oPSLqu1Hf2hMPlTn0HoGQb 2yAottMYmfEhlRLQtlytBW4Abeq14H1qug94t98gfsacqjowUeb5.GJiyKjrS.KuJcn60qY7E4Qj 9Cmp7myx61PmZ2GkBFyXNVmMCqbDqcySPlFBdW8wCww0sze71KXj0t2eLJtHCSfsupzEDv4OH.uG MkndTfl86TN_uVkuD1LWM1ypw2t1HNkd1OUjHQ4nBe79AKpqqZrqf5VbbMn93YBCdwvLr6eZzAvj IH1nVFZKz_47br5zEWW76TILlgbl9rmjiI9jZ9hzK.NvgUTiNpSfLvHP6jSTa8AbHWpkw0uYKVLb 5BaLHXg.8Nhb9IE.ZE_wT0Zv_d7y0KuZZd8r6K.ZldeTaV0aulvw0GkIJdbd.Og9MPUltBW5SkpS h.YMSh8Gb5IxuetiXjDZB67ZzcBPjvUWQVeaOtJl80CZX2BT7ZOdYWPGxHQ.a X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Sat, 10 Jul 2021 20:49:27 +0000 Received: by kubenode538.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 81f24faee1ce70537a7f524a75649ce4; Sat, 10 Jul 2021 20:49:23 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: Too many pythons in poudriere In-Reply-To: <20210710174010.GA91320@www.zefox.net> Date: Sat, 10 Jul 2021 13:49:21 -0700 Cc: freebsd-ports@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <044A7E63-2734-41F4-A1A2-AE5096C6A62C@yahoo.com> <20210708154558.GB60914@www.zefox.net> <20210708175436.GA70414@www.zefox.net> <24795168-CEBF-4049-A4BD-D529F70E7F8D@yahoo.com> <6D9BAA86-CB85-42E9-B4FC-6272141CFFDB@yahoo.com> <20210709230738.GA83522@www.zefox.net> <0313838D-C54A-4517-A53B-C70B4EDDE296@yahoo.com> <20210710174010.GA91320@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GMhwS5CZVz4vRM X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-10, at 10:40, bob prohaska wrote: >=20 > On Fri, Jul 09, 2021 at 05:12:57PM -0700, Mark Millard wrote: >> On 2021-Jul-9, at 16:07, bob prohaska wrote: > [big snip]=20 >=20 >=20 >=20 >>> Just to be clear, after updating kernel and world I gather the=20 >>> suggestion is to repeat >>>=20 >>> # cd /usr/src >>> # make installworld DESTDIR=3D/usr/local/poudriere/poudriere-system = DB_FROM_SRC=3D1 >>> # make distrib-dirs DESTDIR=3D/usr/local/poudriere/poudriere-system = DB_FROM_SRC=3D1 >>> # make distribution DESTDIR=3D/usr/local/poudriere/poudriere-system = DB_FROM_SRC=3D1 >>=20 >> I though you had already synchronized = /usr/local/poudriere/poudriere-system >> to match /usr/src/ and the host root file system in use. If so, no = reason >> to repeat until /usr/src/ (and the booted host root file system) is = updated >> for some reason. Avoid mixes where /usr/src/ is mismatched with the = root >> file system at the time. >>=20 >> Of course, you might want to do another update to those for all I = know. >>=20 >=20 > I tend to update /usr/src and /usr/ports somewhat randomly, based on > elapsed time or in response to trouble of some kind. It looks like = that > habit may be the source of at least some of my trouble.=20 >=20 >> One thing you did not show was the command for after the above block = of >> 4 commands for updates: >>=20 >> # make -DBATCH_DELETE_OLD_FILES delete-old = DESTDIR=3D/usr/local/poudriere/poudriere-system DB_FROM_SRC=3D1 >>=20 >> If you have not done that yet, it would be appropriate. >>=20 >> For the current context of starting over, another command that >> is appropriate (that would normally be delayed to after all the >> poudreire bulk commands are done [were prior package builds >> were instead being put to use]) is: >>=20 >> # make -DBATCH_DELETE_OLD_FILES delete-old-libs = DESTDIR=3D/usr/local/poudriere/poudriere-system DB_FROM_SRC=3D1 >>=20 > I got burned by a delete-old-something long ago, broke most of the = installed ports, and haven't > touched it since. In this case it does seem appropriate. Added to the = update to-do > list. delete-old-libs should only be used after the ports/packages have been updated and installed --unless one is starting from scratch anyway. >> I'll note that the host updating procedure also involves using >> delete-old and, after package updates are all installed, >> delete-old-libs . (No explicit DESTDIR=3D or DB_FROM_SRC=3D use >> involved for the root file system update.) >>=20 >=20 > It resembles the command that bit me.... My guess is that you ran delete-old-libs before having installed updated packages/ports. Having old libraries can lead to its own problems if they continue to be used in places instead of those places updating to the newer library versions. >>> Next,=20 >>> cd /usr/local/poudriere >>> poudriere jail -d main -C all >>> to get rid of the old jail and packages, then re-make the jail with=20= >>>=20 >>> # poudriere ports -c -m null -M /usr/ports >>> # poudriere jail -c -j main -m null -M = /usr/local/poudriere/poudriere-system -S /usr/src -v 14.0-CURRENT >>>=20 >>> Have I got this right? One thing I'm hesitant about is=20 >>> the -C all option.=20 >>=20 >> No reason to delete and recreate the jail, just do a bulk build that >> rebuilds the ports. The first "poudriere bulk" of a sequence can use >> the "-c" option to clear out the existing packages (and logs) before >> the builders start building. >>=20 >=20 > Aye, there's the rub.... It appears that incremental updates are a > costly approach (world/kernel, then a port, then world/kernel...). > It seemed one could get away with it using make in the ports tree, > though perhaps some of my troubles with that approach had the same > origin. =20 You are still trying to get back to a correct environment so things are not normal yet. Plus you are trying to build things that are too big for the environment you are using to just automatically handle it. So you are having to manage the resource use explicitly as you go. [I do not do that: so far I've used systems that have enough resources to allow ALLOW_MAKE_JOBS=3D and all cpus(/cores) to be used by each builder for the things I build, devel/llvm*'s tending to be the largest builds but two or more of those at a time has happened. I'd likely need adjustments for whatever the biggest port builds are.] Use of -c is a very rare thing for me. But when your /usr/src/ goes from something like 1400024 to 1400025, poudriere should notice and to a full rebuild automatically. (So the in world installed in /usr/local/poudriere/poudriere-system better be a match to the /usr/src/ at the time.) The reason for the full rebuild is that the type of change might require it. There is no separate, more detailed, indication about what actually needs to be built. Anyway, poudriere will fully rebuild on a regular basis during periods where that number is changing on a regular basis. main tends to have such changes on a fairly regular basis. >> As far as I know you eventually want to install: >>=20 >> # pkg install chromium >>=20 >> (that will also install things required to run chromium). >>=20 >> I'm not sure if you have a list of other packages that >> you want explicitly installed. So my notes below need >> not cover everything that you want to do. >>=20 >=20 > Getting a running version of chromium was the origninal goal. > The train seems to have run off the tracks......8-) >=20 >> (I do not know if ALLOW_MAKE_JOBS=3D and such is appropriate for the >> llvm10 bulk build but it may fit in the resources okay:) >>=20 >> # poudriere bulk -j main -c devel/llvm10 > = bulk_from_scratch_for_llvm10_1st.log >>=20 >> (I'm unclear if you would want a -J? to limit the builders >> count for the above command: if so, use it too.) >>=20 >> That -c will first clear out the existing package builds (and >> the logs according to the documentation). Then it will start >> from scratch building the required ports. >>=20 >> This will build a lot of ports, those needed to in turn build >> devel/llvm10 . So having -J1 used and ALLOW_MAKE_JOBS missing >> would make for a very long build time. You probably want to >> allow some combination of multiple cores to be in use, >> possibly also involving MAKE_JOBS_NUMBER=3D use. You know more >> about what combinations worked before. >>=20 >> Reminder that ALLOW_MAKE_JOBS=3D control would go in: >>=20 >> /usr/local/etc/poudriere.conf >>=20 >> but MAKE_JOBS_NUMBER=3D control would go in: >>=20 >> /usr/local/etc/poudriere.d/make.conf >>=20 >=20 > That is a helpful distinction unclear to me till now. >=20 >> (I do not know if ALLOW_MAKE_JOBS=3D and such is appropriate for the >> rust bulk build but some combination may fit in the resources okay:) >>=20 >> # poudriere bulk -j main lang/rust > bulk_for_rust_2nd.log >>=20 >> (I'm unclear if you would want a -J? to limit the builders >> count for the above command: if so, use it too. You know >> more about what combinations worked before.) >>=20 >> (Without ALLOW_MAKE_JOBS=3D is likely appropriate --or at least the >> MAKE_JOBS_NUMBER=3D may be needed-- for:) >>=20 >> # poudriere bulk -j main www/chromium > bulk_for_chromium_3rd.log >>=20 >> (I'm unclear if you would want a -J? to limit the builders >> count for the above command: if so, use it too. You know >> more about what combinations worked before.) >>=20 >> Note that the management of the limited RPi3B context involves >> adjusting things like ALLOW_MAKE_JOBS=3D and -J? usage for each >> huge build to limit resource use to an appropriate scale for >> those builds. For smaller builds, more can be allowed (more >> adjustments). >=20 > Since I usually play with ports one at a time, is there any > advantage to using testport versus bulk? Better diagnostics, > maybe?=20 man poudriere-testport: The specified port will be tested for build and packaging problems. = All missing dependencies will first be built in parallel. = TRYBROKEN=3Dyes is automatically defined in the environment to test ports marked as = BROKEN. See FLAVORS in poudriere(8) for supported FLAVORS syntax.\ but has a longer description in man poudriere. I'm going to quote that about what I think is different from bulk vs. the same as bulk when a single port is listed on the poudriere command line: QUOTE [with notes] Test a single port This second example show how to use poudriere for a single port. = Take the example of building a single port; poudriere testport -o category/port -j myjail all the tests will be done in myjail. It starts the jail, then mount the ports tree (nullfs), then mounts = the package dir (poudriere/data/packages/--), = then it mounts the ~/ports-cvs/mybeautifulporttotest (nullfs) it builds = all the dependencies (except runtime ones) and log it to = poudriere/data/logs/testport/jailname/default/mybeautifulporttotest.log). [Ignore the ~/ports-cvs/mybeautifulporttotest detail from what I can = tell. However, I believe this is indicating that the places that some things = would go for normal use will not be used. Some temporary place is apparently used instead and the material in that temporary place is not kept long = term.] If packages for the dependencies already exist, then poudriere will = use them. [Other than a temporary spot vs. the normal place, this is like bulk.] When all the dependencies are built, packages for them are created = so that next time it will be faster. [This does indicate that, like bulk, the dependencies will be remembered and reused when built. That does not say that the port being tested will be remembered or resued, however.] All the dependency phase is done with PREFIX =3D=3D LOCALBASE. After that it will build the port itself with LOCALBASE !=3D PREFIX [There is a testport option -P for "Use custom prefix".] [bulk would use the normal PREFIX =3D=3D LOCALBASE unless the port itself makes assignments. This is a test for if the port is handling PREFIX and LOCALBASE correctly. The earlier ~/ports-cvs/mybeautifulporttotest material is possibly trying to reference to where the unusual place PREFIX might be. But I see nothing in the scripts constructing that kind of path automatically.] and log the build to = poudriere/data/logs/testport/jailname/default/mybeautifulporttotest.log [Again ignore the "mybeautifulporttotest.log" naming detail. bulk would not put the log file under poudriere/data/logs/testport/jailname/default/ .] Poudriere will try to: install it, create a package from it, = deinstall it, check for cruft left behind and propose the line to add to = pkg-plist if needed. [bulk would not do such an install/create package from = installed/deinstall in some temporary place to test the installation/deinstallation = handling.] END QUOTE I do not see any differences there that helps in your build activities or for better error/warning messages about the types of errors that you get. I'd just use bulk, listing one port on the command line, not testport . (I do not use testport because I'm not developing ports or significant changes to port Makefiles and such.) Using testport does not change needing to manage the resource use for devel/llvm10 lang/rust www/chromium steps on the RPi3B. Nothing will get rid of having to do that management for building on a RPi3B. The only thing that will get rid of needing to the that management is to use something other than a RPi3B that has more resources (especially RAM in proportion to core counts). >> I do not know if there are other ports that you would want >> to build and later install that are not automatically built >> by the above or if you have a list of such in a file that >> can be used with -f . In your context this would probably >> be a list of ports that do not involve huge builds, given >> that you had already done the above bulk runs (and any others >> that might involve more huge builds). >>=20 >=20 > It's clear that by segregating build dependencies from run > dependencies poudriere provides a valuable service. Might > there be a make target in ports that cleans up no-longer-needed > build dependencies during installation of a given port? That > would, I think, have been enough to avoid the python27 conflicts. How many other ports might have the same build dependency? How many other ports might have a run dependency on the same port that is just a build dependency of the port in question? How many other ports would be damaged by the removal of what the specific port has as only a build dependency? Rebuilding each dependency each time a port is built would use massive time and resources (fully independent builds). poudriere manages to reuse built ports in more contexts instead of rebuilding them but keeps things clean anyway. Dependecy management is not a local-to-port thing but a more global thing, unfortunately. After poudriere has everything built for later installation: # pkg delete -a . . . # pkg install chromium . . . will remove all the ports (except pkg) and then install just the ones needed for chromium's operation. > Is there a plain-language description of the logic poudriere follows > for building ports? I found this: > = https://forums.freebsd.org/threads/pkg-package-repository-using-ports-mgmt= -poudriere-with-or-without-zfs.38859/ > but it doesn't really touch on the topics I'm looking for. Something > on the level of a simplified flowchart or (gasp!) PowerPoint, maybe?=20= Out of all the things involved in poudriere's operation I've no clue how to identify just the possibly small subset you may be curious about. I doubt that you want to learn about all aspects of poudriere's operation. (Well, beyond my range of knowledge. I investigate as needed as I go along.) What you sight is (old) how-to-use documentation, not how-does-it-work documentation. So I'm not even sure which you are after. Either way, I'm not aware of anything that is a likely match to whatever you are hoping to see. A lot of your questions are of the nature "how do I side step the available commands for how to use poudriere". "Plain-language" and jails and repositories with transitional updates (all updates completed vs. no updates being the two results, no mixed results) and such do not tend to go together. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)