Date: Mon, 28 Dec 2020 09:29:52 -0700 From: "Edward Sanford Sutton, III" <mirror176@hotmail.com> To: freebsd-ports@freebsd.org Subject: Re: portmaster new development Message-ID: <DM6PR03MB3674E71A2C94E637FFC7C083E6D90@DM6PR03MB3674.namprd03.prod.outlook.com> In-Reply-To: <20201227060021.3eb4148d@dismail.de> References: <20201226124150.7c494410@dismail.de> <6d0d128b-9a75-34f4-830c-d8be05ded9cb@freebsd.org> <X%2BhEUk7K92mi%2BEca@home.opsec.eu> <X%2BhMcRqUX17XyyBQ@c720-r368166.fritz.box> <2128EDF2-39D7-4FF4-92AB-9F36FA4CAAC5@ellael.org> <20201227060021.3eb4148d@dismail.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/27/20 4:00 AM, LuMiWa via freebsd-ports wrote: > The subject is 'portmaster new development' but again start pushing > poudriere to FreeBSD users. I do not use zfs file system and I do not > use poudriere and I do not want to use on my computer for building some > ports and then spending hours and hours with poudriere with not enough > machine. For me is portmaster perfect as is now. More tools and improvements to tools can be a good thing. There are some things that portmaster did better and others that portupgrade did better. Can't remember which for each task but I think I used portmaster for configure dialogs and fetching as it would perform those steps before builds started. I found myself using portmaster to create a list of ports I had installed (portmaster --list-origins > ~/installed-port-list) to then feed into poudriere as a list to build though pkg prime-origins completes in a small fraction of a second instead of more than 5 seconds. I've dealt with a number of ports that fail to build because of a conflict (usually not noted). May have been avahi or a different port sounding similar in my head but one would regularly fail to upgrade for me if it was currently installed because the build found and used tools from the old version during the build of the new even after it spent time building the new tools in the work directory. People probably think those issues are rare in the ports tree but back around 2009-2011 I upgraded a x86 to x64 by upgrading in place. Running portupgrade lead to many failures due to ports using locally installed 32bit tools and libraries even though in theory everything should have been rebuilt/reinstalled in an orderly fashion where it would be replaced before it gets used. I've thought about repeating that to document findings but unless other developers want to look into the issues then there is no point as I have not been able to fix those type of build issues when I've tried. Deleting any installed package before its own port rebuild would work around many of those issues I saw but you really need to uninstall all nondependent ports to guarantee an individual port build succeeds. If uninstalling a port does not remove everything (bad pkg-plist) it created then there is still a chance of a contaminated build. Some builds will break if you have nondependencies installed and other times it may just modify what you get from building it as the program detects its surroundings and makes findings the port maintainer didn't account for when the build autodetects surroundings with things like autotools which didn't have the maintainer override or add a dependency for every possible optional manual detection. Remove port(s), rebuild, and reinstall once built is higher downtime than a pkg upgrade of already built ports. That is made worse when portmaster aborts on an unrelated dependency tree branch causing a complete build halt mid-upgrade. Portupgrade's "continue building all others on fail" leaves less programs broken without interaction but still higher than if it rolled back the entire dependency tree of a failed port. Building in a jail gets around the issue of needing a separate build machine, removing installed packages, and having program downtime while waiting for the chain of it+dependencies to all be rebuilt (incomplete build of program+dependencies leads to broken programs if I cannot resolve). That time would be extended when I would have days worth of building to do and portmaster would stop building within hours or minutes of starting and I won't see results for hours later when I get home from work or wake up. I completed very little with portmaster before going back to portupgrade back in the day because abort on failure rarely errored out on the first or last of a dependency chain and stopping in the middle without rolling back the entire chain would mean the update attempt generally broke things other than the one failed port. Installing from packages after all building is done offers the shortest downtime of programs on the system. A poudriere build stays in its own area and keeps it clean as it goes so those problems go away. Other solutions including a portmaster/portupgrade building in a jail should be able to do so too. Poudriere can be configured to not use excessive resources by controlling number of different ports to build at once, how many threads to use per port build, and how much to use memory disks vs real disks for workdirs, ccache, etc. Poudriere will take longer due to always starting with a clean environment, being more aggressive about deciding when a port needs a rebuild, and time spent building/updating the base system to be used inside the jail. Other jailed build systems will likely perform faster due to less cleanup per package. Poudriere defaults performing a full build for my machine while I use my machine is a bad experience but had controls to adjust settings to all be more sane. Poudriere, with more tame settings than default, leads to a bit longer build time for much less downtime on my desktop with CPU from 2012(?) which does both the building and normal desktop use. It does all this on spinning rust as I haven't taken up the task of various filesystem changes I want to make as I upgrade to SSDs even though I've been talking customers into (and forced family into) SSD upgrades for years. Poudriere still offers the fine grained + general changes to customize/break builds. Last changes I made were trying to enable options for every port checking them with OPENMP breaking imagemagick builds and FFMPEG causing problems. I took out unsetting NLS; can't remember why but probably sorting out problems too; reason for using it was I only speak English and additional dependencies for unused language capability was just more bloat likely to slow building more and lead to more rebuilds when the seemingly unneeded dependency gets an update. I'm playing with turning on "DTRACE THREADS OPTIMIZED_FLAGS OPTIMIZED_CFLAGS OPENCL ASM MMX SSE SSE2 SSE3 SSE41 SSE42 AVX" though wouldn't be surprised to see it is why some videos don't play properly in VLC now, or could be options I need to review but haven't checked. The more you change builds from defaults, the more likely you are to find issues first and not all show up in mailing lists presuming you even notice its a custom issue as it may just be less stable software instead of a build failing with an error message to pass to google. The following was a command I built up over time to help download distfiles before running portupgrade. Exclusions were to keep old versions, keep things not in the ports tree (deleted ports, ports I worked on but never reached a point of submission, etc.), and ignore things otherwise causing errors. If doing the same thing, you can build this command one -x at a time but it gets slower waiting for each iteration to reach an error as you have to start at the beginning on every abort which by the end was >10 minutes to just process through the list of ports to the point of the next failure so takes many hours to create this command. End result is portmaster was nice to see but a big waste of time for me though didn't stop me from trying and I do look forward to upgrades of it even if I try to not use such tools now. portmaster -af -x cinelerra -x replicatorg-0025 -x mp3encode-1.10_2 -x electricsheep-2.6.8_7,1 -x ltmdm-1.4_19 -x uppc-kmod-0.8_3 -x libcapplet-1.4.0.5_11 -x autopsy-2.08 -x festlex-ogi-2.2_1 -x mjpegtools-uvfilters-1.1_4 -x p5-Getopt-Mixed-1.10 -x wine-1.1.36,1 -x zsnes-1.51_4,1 -x sphinx-2.0.4_2 -x wavplay-1.4_2 -x gogo-2.39b_1 -x svgalib -x win32-codecs-3.1.0.r1,1 -x inventor-2.1.5.p10_5 -x linux-f10-xorg-libs-7.4_1 -x xmlto-0.0.23 -x system-config-printer-1.0.16 -x gnome-user-docs-2.28.2 -x gtkhtml-1.1.10_11 -x ocaml-lablgtk2-2.14.0 -x mldonkey-3.0.1_1 -x portdowngrade-0.6_1 -x strace-4.5.18_1 -x teamspeak_server-2.0.24.01 -x zgv-5.9_2 -x bonic-einsteinathome-4.18_2,2 -x cpuburn-1.4 -x bladeenc-0.94.2 -x emerald-themes-0.5.2 -x libcapsinetwork-0.3.0 -x gcc-4.4.7.20110628 -x monopd-0.9.3 -x koffice-1.6.3_17,2 -x avr-gcc-4.5.1 -x libmusicbrainz-2.1.5 -x kdelibs-3.5.10_8 -x libtunepimp-0.5.3_9,1 -x p5-Version-Requirements-0.101021 -x freepops-0.2.9_2 -x gpgme-0.3.16_8 -x p5-HTTP-Message-6.02 -x gcc-4.4.7.20120117 -x kde-icons-noia-warm-0.95 -x gcc-4.4.7.20120306 -x cgal-3.6.1 -x k3b-1.0.5_13 -x kdegraphics-kuickshow-3.5.10_3 -x mjpegtools-2.0.0_2 -x fnlib-0.5_9 -x libmapi-0.9_2 -x optipng-0.6.5 -x fuser-1142334561_3 -x portmanager-0.4.1_9 -x freetype-1.3.1_4 -x libutempter-1.1.5_1 -x gcc-4.4.7,1 -x avogadro-1.0.3_1 -x evas-engine-xrender-1.0.1 -x libxml-1.8.17_5 -x ja-vflib-2.25.6_12 -x jdk-1.6.0.3p4_29 -x kde4-freebsd-carddeck-1.0 -x linux-realplayer-10.0.9.809.20070726_3 -x py27-clutter-1.0.2_3 -x xmountains-2.9 -x gtkglarea-1.2.3_3 -x kphotoalbum-3.1.1_10 -x teamspeak_client-2.0.32_2 -x xloadimage-4.1.17_1 -x kde-3.5.10_8 -x kdeaccessibility-3.5.10_5 -x kdeaddons-3.5.10_5 -x kdeaddons-atlantikdesigner-3.5.10_4 -x kdeaddon s-kaddressbook-plugins-3.5.10_6 -x kdeaddons-kate-plugins-3.5.10_6 -x kdeaddons-kfile-plugins-3.5.10_6 -x kdeaddons-kicker-applets-3.5.10_6 -x kdeaddons-knewsticker-scripts-3.5.10_4 -x kdeaddons-konq-plugins-3.5.10_6 -x kdeaddons-ksig-3.5.10_4 -x kdeaddons-noatun-plugins-3.5.10_4 -x kdeaddons-renamedlg-plugins-3.5.10_4 -x kdeadmin-3.5.10_4 -x kdeartwork-3.5.10_6 -x kdeartwork-xscreensaver-kde-3.5.10_7 -x kdebase-3.5.10_12 -x kdebase-kompmgr-3.5.10_4 -x kdeedu-3.5.10_5 -x kdegames-3.5.10_4 -x kdegraphics-3.5.10_11 -x kdegraphics-kamera-3.5.10_4 -x kdegraphics-kooka-3.5.10_4 -x kdegraphics-kuickshow-3.5.10_4 -x kdelibs-3.5.10_11 -x kdemultimedia-mpeglib_artsplug-3.5.10_6 -x kdemultimedia-xine_artsplugin-3.5.10_8 -x kdenetwork-3.5.10_4 -x kdenetwork-kopete-0.12.8_7 -x kdepim-3.5.10_12 -x kdesdk-3.5.10_8 -x kdetoys-3.5.10_4 -x kdeutils-3.5.10_10 -x kdewebdev-3.5.10_9,2 -x dbus-qt3-0.70_6 -x qt-copy-3.3.8_14 -x py27-qt-3.18.2_2,2 -x akode-plugins-mpc-2.0.2,1 -x akode-plugins-oss-2.0.2,1 -x arts-1.5.10_8,1 -x artswrapper-1.5.3 -x kde-xdg-env-1.0_3,1 -x kdehier-1.0_11 -x akode-2.0.2_2,1 -x akode-plugins-ffmpeg-2.0.2_4,1 -x akode-plugins-jack-2.0.2,1 -x akode-plugins-mpeg-2.0.2,1 -x akode-plugins-resampler-2.0.2,1 -x akode-plugins-xiph-2.0.2_4,1 -x k3b-1.0.5_16 -x kde-workspace-4.8.4 -x kdemultimedia-3.5.10_7 -x libkdcraw-0.1.9_5 -x libkexiv2-0.1.9_8 -x libkipi-0.1.6_6 -x qca-tls-1.0_8 -x qscintilla-1.7.1_8 -x digikam-0.9.6_4 -x digikam-doc-0.9.5_4 -x dolphin-0.8.2_6 -x filelight-1.0_7 -x kmplayer-0.10.0c_8,2 -x imgSeek-0.8.6_15 -x potracegui-1.3.4_7 -x kmldonkey-0.11_4 -x klibido-0.2.5_9 -x krename-3.0.14_6 -x kipi-plugins-0.1.7_11,1 -x kdirstat-2.4.4_7 -x qcad-2.0.5.0.1_8 -x krusader-1.90.0_6 -x libkexif-0.2.5_10 -x qsa-1.1.5_1 -x linux-realplayer-10.0.9.809.20070726_3
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DM6PR03MB3674E71A2C94E637FFC7C083E6D90>