From owner-freebsd-ports@freebsd.org Sun Dec 27 12:54:17 2020 Return-Path: Delivered-To: freebsd-ports@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF1494BAD2D for ; Sun, 27 Dec 2020 12:54:17 +0000 (UTC) (envelope-from se@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D3gc96759z4blZ; Sun, 27 Dec 2020 12:54:17 +0000 (UTC) (envelope-from se@freebsd.org) Received: from Stefans-MBP-WLAN.fritz.box (p200300cd5f2f8c00345ecbf25931f219.dip0.t-ipconnect.de [IPv6:2003:cd:5f2f:8c00:345e:cbf2:5931:f219]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 5DB6D2EDB; Sun, 27 Dec 2020 12:54:17 +0000 (UTC) (envelope-from se@freebsd.org) To: Thomas Mueller , freebsd-ports@freebsd.org References: <20201226124150.7c494410@dismail.de> <6d0d128b-9a75-34f4-830c-d8be05ded9cb@freebsd.org> <20201227050406.61CC2ECC@StefanEsser.freebsd.org> From: Stefan Esser Subject: Re: portmaster new development Message-ID: <734709a9-c989-56ee-45f3-ed2cc8cd0335@freebsd.org> Date: Sun, 27 Dec 2020 13:54:14 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <20201227050406.61CC2ECC@StefanEsser.freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="rSoQnoOriZofoyBqnkWq6tngL7U8oxDjs" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Dec 2020 12:54:18 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --rSoQnoOriZofoyBqnkWq6tngL7U8oxDjs Content-Type: multipart/mixed; boundary="AhUUBxVJZnRVaZcjboyqvfaPeDWs8Qu2r"; protected-headers="v1" From: Stefan Esser To: Thomas Mueller , freebsd-ports@freebsd.org Message-ID: <734709a9-c989-56ee-45f3-ed2cc8cd0335@freebsd.org> Subject: Re: portmaster new development References: <20201226124150.7c494410@dismail.de> <6d0d128b-9a75-34f4-830c-d8be05ded9cb@freebsd.org> <20201227050406.61CC2ECC@StefanEsser.freebsd.org> In-Reply-To: <20201227050406.61CC2ECC@StefanEsser.freebsd.org> --AhUUBxVJZnRVaZcjboyqvfaPeDWs8Qu2r Content-Type: multipart/mixed; boundary="------------245C848944233E4C8639DC35" Content-Language: en-US This is a multi-part message in MIME format. --------------245C848944233E4C8639DC35 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Am 27.12.20 um 06:04 schrieb Thomas Mueller: >>> And as portsnap user I have a question: Do they planning deprecation = of >=20 >>> portmaster too? >=20 >> No, I'm actively working on portmaster and have rewritten it from >> scratch for better performance (and additional features, e.g. building= >> in a clean chroot jail, similar to synth or poudriere). >=20 >> I have been using that version for more than one year, but the >> functionality is not complete, yet. >=20 >> On a test system with > 2200 installed ports it takes less than 10 >> seconds to identify the ~600 out-of-date ports (that I keep in this >> state for testing of the upgrade strategy function), which is more >> than 30 times faster than the same operation with the "official" >> portmaster. >=20 >> Until completion of that version, I'll continue to maintain and >> update the current portmaster port ... >=20 >> Regards, STefan >=20 > Question about the relation of portsnap and portmaster reminds me of Ja= va and Javascript, or potato and sweet potato (not closely related). Yes, portsnap and portmaster do not share anything beyond the first 4=20 letters of their names ... > Since my question is about a new portmaster, I rename the subject to "p= ortmaster" or "portmaster new development", rather than hijack the "ports= nap" thread. >=20 > Which portmaster do I get if I build and install what is currently in t= he ports tree? This is a version that I became maintainer of when it lacked flavor=20 support and the original developer (Doug Barton) had left the project. > amelia2# ls -l ports-mgmt/portmaster > total 16 > -rw-r--r-- 1 root wheel 1479 Dec 27 02:01 Makefile > -rw-r--r-- 1 root wheel 184 Feb 28 2018 distinfo > drwxr-xr-x 2 root wheel 512 Dec 27 02:01 files > -rw-r--r-- 1 root wheel 1189 May 6 2019 pkg-descr >=20 > from a fresh svn up of the ports tree. Yes, I added a feature requested by Kevin Obermann, yesterday, who had noticed that a locked port could be built but not installed, if the user answers "yes" to the question whether the lock should be ignored. > An improved portmaster arouses my interest. Maybe modify the name so i= t can be added to the ports tree and coexist with the "official" portmast= er. After taking over the current portmaster I noticed that it is ancient in its structure. It used global variables throughout and forked a lot if sub-processes to allow for local state. Since it is extremely cumbersom to work on a linear 4000 line shell script, I re-wrote it from scratch using bash associative arrays and used that version since the summer of 2019. That version did already support clean builds in a chroot jail as an option. But I noticed that better data structures would allow to substantially speed-up the scanning phase and decided to port that bash based version to LUA. It can be found in my GH repo, but I'm currently reworking the planning phase to allow for multiple pre-decessors of a port (e.g. if A-v1 is integrated into B-v2 and B-v1 is already installed). My current (not pushed) development version locks up due to such a case (deskutils/kdepim-apps-libs has been merged into net/akonadi-contacts and akonadi-contacts is downlevel, leading two an attempt to get an exclusive lock on the target package name twice, since the two update tasks are not currently merged into one). But other than that it has been usable for simple updates for a long time already. > Desired features/options would be to keep going rather than stop when o= ne port fails to build, and the ability to install build dependencies, wh= ich may be useful for building other software. Yes, I mark failed updates as such and cancel planned updates of dependencies, but unrelated updates go on. > With synth, I had a difficult time getting everything that was built to= install, some packages like bison are needed in building other software.= The bash version supported 4 modes: traditional portmaster, delayed installation of packages not depended on in the update run, jailed builds with installation of packages after completion of all builds and repository mode to just create packages in a jail. I ran into issues when I tried to optimize the disk space used for the jail by de-installing no longer required build dependencies from the jail as soon as possible. I needed better data structures than offered by bash to efficiently express these dependencies. That was the point where I decided to migrate the code to LUA. > How is poudriere in that regard? I never used poudriere, have been int= imidated by not wanting to use zfs or dialog4ports, or such an elaborate = setup just to update one or a few ports. I have to use poudriere to test ports before commit, but I do not like it. The build jails are quickly becoming stale and have to be rebuilt (causing all previously built packages to be compiled again) and I often run into issues where a build dependency fails for reasons that ar enot obvious (e.g. gmake) and I have found that the easiest method of recovery is to throw away the whole poudriere jail and to start over with a fresh one. > Gentoo Linux with portage has "--with-bdeps=3Dy" which installs build d= ependencies when desired. Poudriere builds every port in a clean jail with dependencies installed from previously built packages. This jail is destroyed when the new package has been saved and then another fresh one is created for the next port to build. This process takes advantage of ZFS features (when building on ZFS, as suggested) and it uses cloned file systems to quickly create each jail in a known sane state. > I found that poudriere uses dialog4ports; I much prefer to save options= in a file such as Gentoo Linux does with make.conf and (NetBSD) pkgsrc d= oes with mk.conf . You always can do this - and poudriere is generally used with options provied in jail-specific make.conf files. > I once got a royal mess of circular/jumbled dependencies with dialog4po= rts; cleaning was a major nuisance, nothing simple like editing /etc/mk.c= onf or /etc/make.conf . This should not happen and should be reported to the port maintainers to look into. > I would like to be free of dialog4ports; the older dialog was worse and= messed up my screen. Yes, I'm often working remotely on a MacBook and have to find a TERM description that works well with the options dialog. I have appended a "port" that builds the LUA version I'm working on as of 2 months ago. Recent changes to rework the planner to deal with the above mentioned issue has broken previously working features and I have not pushed any later commit to keep the head version in Git usable. That version has incomplete dependency tracking and only supports variants of "portmaster -a" (especially not "portmaster " as it does not reliably collect all dependencies in that case). This version has the lock-up issue described above, but should be usable otherwise. (The lock-up can be worked around by deleting the kdepim-apps-libs package before starting portmaster, but I keep my test system in this state to easily be able to test the new planner that shall be able to deal with this situation.) I'd be interested in system configurations and the times reported by $ time portmaster.lua --no-confirm -n -a This takes less than 10 seconds real time on my test system which has a Ryzen 5 3600 CPU and ZFS with a L2ARC on SSD with 2200 ports and 600 of them being downlevel. But I do not have performance data from significantly different systems ... (The above command may start to fetch missing dist files. I'm interested in the times for a system that already has all required distfiles available.) To display the operations that would be performed you can use: $ portmaster.lua --dry-run -a Regards, STefan --------------245C848944233E4C8639DC35-- --AhUUBxVJZnRVaZcjboyqvfaPeDWs8Qu2r-- --rSoQnoOriZofoyBqnkWq6tngL7U8oxDjs Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEo3HqZZwL7MgrcVMTR+u171r99UQFAl/og/YFAwAAAAAACgkQR+u171r99UR2 wwf9EkzoRs11DVlYe7PS346atZXi8cmcVRzNG33yQNfcWk4sVCaQRCXip5TbxsSHT6YlLH/m1LRb LyRQ/ndTJr9hpxchagGya4pwzdkS+GXUv5P+1msXXRerOioS6acJCxCFL4HlLxcQ3sln+8lPAyj+ bz9fo8gBrno9DDGM/ttDGxzT0JfyGwx8rY3ax1YW6/MZwToG2G/tpc3KdnDo9RkQQWAn8E8tGUXr QPtbTBzNiRLFhgzDJvVZ/9Qv/bi8RZpMkRfkdqacEwvDZAvIOqfDE4JPh9YcYUXLDEQvHI4j0h06 /FlZlgw5hKh4gODSzCOmnfyiMNIzSB6duvALMbYx2Q== =Ru8F -----END PGP SIGNATURE----- --rSoQnoOriZofoyBqnkWq6tngL7U8oxDjs--