Date: Wed, 27 Apr 2016 15:40:27 -0700 From: Chuck Outman <charlesoutman@yahoo.com> To: freebsd-ports@freebsd.org Message-ID: <407950.49499.bm@smtp238.mail.gq1.yahoo.com>
next in thread | raw e-mail | index | archive | help
ClNlbnQgZnJvbSBteSBWZXJpem9uIDRHIExURSBEcm9pZA== From owner-freebsd-ports@freebsd.org Thu Apr 28 00:17:17 2016 Return-Path: <owner-freebsd-ports@freebsd.org> Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF7FEB1D985 for <freebsd-ports@mailman.ysv.freebsd.org>; Thu, 28 Apr 2016 00:17:17 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id CF6F211B3 for <freebsd-ports@freebsd.org>; Thu, 28 Apr 2016 00:17:17 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id CB086B1D983; Thu, 28 Apr 2016 00:17:17 +0000 (UTC) Delivered-To: ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAA7EB1D982 for <ports@mailman.ysv.freebsd.org>; Thu, 28 Apr 2016 00:17:17 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7628811B2 for <ports@freebsd.org>; Thu, 28 Apr 2016 00:17:17 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id u3S0H43A005409; Wed, 27 Apr 2016 17:17:09 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201604280017.u3S0H43A005409@gw.catspoiler.org> Date: Wed, 27 Apr 2016 17:17:04 -0700 (PDT) From: Don Lewis <truckman@FreeBSD.org> Subject: Re: Ports tree gone unstable? To: michelle@sorbs.net cc: ports@freebsd.org, vmiller@hostileadmin.com, rkoberman@gmail.com In-Reply-To: <572140C0.1030903@sorbs.net> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Porting software to FreeBSD <freebsd-ports.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-ports>, <mailto:freebsd-ports-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-ports/>; List-Post: <mailto:freebsd-ports@freebsd.org> List-Help: <mailto:freebsd-ports-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-ports>, <mailto:freebsd-ports-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 28 Apr 2016 00:17:18 -0000 On 28 Apr, Michelle Sullivan wrote: > Don Lewis wrote: >> On 27 Apr, Michelle Sullivan wrote: >>> Don Lewis wrote: >>>> On 27 Apr, Michelle Sullivan wrote: >>>>> Don Lewis wrote: >>>>>> On 27 Apr, Michelle Sullivan wrote: >>>>>>> Don Lewis wrote: >>>>>>>> On 27 Apr, Rick Miller wrote: >>>>>>>>> On Wed, Apr 27, 2016 at 12:53 PM, Michelle Sullivan <michelle@sorbs.net> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Kevin Oberman wrote: >>>>>>>>>> >>>>>>>>>>> On Wed, Apr 27, 2016 at 8:06 AM, Michelle Sullivan <michelle@sorbs.net> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> After a portsnap update it seems all my jails won't build the current tree >>>>>>>>>>>> returning the following error: >>>>>>>>>>>> >>>>>>>>>>>> ====>> MOVED: sysutils/puppet renamed to sysutils/puppet38 >>>>>>>>>>>> ====>> MOVED: textproc/rubygem-augeas renamed to >>>>>>>>>>>> textproc/rubygem-ruby-augeas >>>>>>>>>>>> >>>>>>>>>>>> ====>> Computing deps for converters/libiconv >>>>>>>>>>>> ====>> Computing deps for archivers/unzip >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> >>>>>>>>>>>> ====>> Computing deps for converters/p5-Encode >>>>>>>>>>>> ====>> Computing deps for converters/p5-Convert-BinHex >>>>>>>>>>>> ====>> Computing deps for converters/p5-Encode-Locale >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Computing deps for converters/p5-JSON-PP >>>>>>>>>>>> ====>> Computing deps for converters/p5-JSON >>>>>>>>>>>> ====>> Computing deps for converters/p5-JSON-XS >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> >>>>>>>>>>>> ====>> Computing deps for converters/p5-Text-Iconv >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Computing deps for databases/ip4r >>>>>>>>>>>> ====>> Computing deps for databases/gdbm >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Computing deps for databases/p5-Bucardo >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> >>>>>>>>>>>> Terminated >>>>>>>>>>>> Terminated >>>>>>>>>>>> Terminated >>>>>>>>>>>> Terminated >>>>>>>>>>>> ====>> Cleaning up >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Computing deps for databases/p5-DBD-Pg >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/ccache' not found. >>>>>>>>>>>> ====>> Computing deps for databases/memcached >>>>>>>>>>>> ====>> Error: Invalid port origin '/usr/local/bin/automake-1.15' not >>>>>>>>>>>> found. >>>>>>>>>>>> ====>> Umounting file systems >>>>>>>>>>>> >>>>>>>>>>>> Checked updating but don't see anything to suggest that port origins of >>>>>>>>>>>> '/usr/local/bin/ccache' are normal.. >>>>>>>>> It looks like you're building with Poudriere. I observed similar behavior, >>>>>>>>> but not the exact message the other day. I don't remember what origin it >>>>>>>>> was complaining about, but located a post (either on a mailing list or >>>>>>>>> forums) recommending a `pkg install poudriere`. It did resolve the problem >>>>>>>>> in this particular scenario. >>>>>>>> This is probably caused by the recent change to globally drop >>>>>>>> ${PORTSDIR} from *_DEPENDS. The framework changes initially were done >>>>>>>> in bsd.port.mk r399278, but the the actual removal of ${PORTSDIR} didn't >>>>>>>> happen until r411970, r412342, ... >>>>>>>> >>>>>>> Ok that sorta makes a bit more sense... however as this is a jail and >>>>>>> the tree is updated why did it break? (I have no local mods in the 'ng' >>>>>>> build tree - except an additional (local only) couple of ports which are >>>>>>> copied in manually after the portsnap update)... >>>>>>> >>>>>>> Of course the nice thing is my non-ng tree is still working 100% - but >>>>>>> that would be because it didn't get the change... but again that's a >>>>>>> completely separate tree and the 2 are not associated with each other in >>>>>>> any way... >>>>>> I was assuming that this was your non-ng tree where you have local >>>>>> framework changes ... >>>>> No, completely separate repo as the new trees are constantly breaking my >>>>> tree so I keep them entirely separate. >>>>>> Did you upgrade ports from something older than r411970 (Sun Mar 27 >>>>>> 01:23:25 2016 UTC) >>>>> It would have been around march I did the last build so yes probably >>>>> prior to Mar 27. >>>>> >>>>>> to something more recent? >>>>> To the latest. >>>>>> If poudriere on your host >>>>>> is seriously old, it might not cope with the framework change. It looks >>>>>> like you need at least 3.1.9, which was released on Wed Oct 14 21:06:00 >>>>>> 2015 UTC. >>>>> Yeah, 3.1.x changes the base OS without authority and breaks the entire >>>>> build system (can't build anything but the official tree in it) so it's >>>>> been deemed a security issue (because it "upgrades" the existing >>>>> repositories) and therefore cannot be installed or used on any of the >>>>> existing build servers. >>>> Sorry, this is all from memory ... my poudriere machine will be offline >>>> for several more hours so I can't use it as a reference. >>>> >>>> Poudriere shouldn't be changing anything in the base OS. It probably >>>> creates some temp files under /tmp and puts the package repositories >>>> that I builds and the log files under its own directories under >>>> /var/tmp. >>> Unfortunately the first thing that 3.1.[012]? did was install all the >>> pkg stuff and change the pkg_add repo into a pkg repo... or something >>> like it, which broke everything horribly.. it was a long time ago so no >>> idea the specifics now... but it (3.1.x) was put on a 'not suitable for >>> use due to security issues' list. >> That could be the point at which support for the old pkg_* tools was >> removed from poudriere. That would explain why it wanted to upgrade >> things. > > Yes, but that's also a security issue in itself. Had I not spotted and > stopped this the changes it tried to make to the repo would have > resulted in production machines being 'reload from scratch' on the first > security patch - because it screwed the build servers and the repos. I would think that the pkg_* tools would see the NG repo as something that they didn't understand and then would barf without doing anything. >> >> I'd thought that poudriere was using the host copy of pkg to do the >> final part of the respository build, but since poudriere doesn't list >> pkg as a dependency, that appears not to be the case. It looks like >> poudriere is running pkg (from the repository being constructed) in the >> jail for that. > > Yeah.. except something around the time went about "upgrading" the OS to > use pkg as well... which screwed the OS... fortunately I caught the > first VM it tried to do it to and was able to limit the damage just to > that VM so the rebuild was minimum. That's quite possible. If the OS is old enough to have been using the old pkg_* tools by default, then it would have gotten an update to switch to pkgNG when the support for old-style packages was removed from the ports tree. Prior to that, there was a lengthy period of time when old packages were the default, but you could switch to new packages by adding some magic to /etc/make.conf. I thought all the magic was in the ports framework and prior to the cutover it just looked at ${OSVERSION}, though. A client without a copy of /usr/ports wouldn't know about any of this. I would think that the old pkg_* tools would still work with an old-style repository, but that's not something that I've tried. >> >>>> You should be able to build as many different ports trees as you want >>>> and they can be downloaded via portsnap or svn, or created by hand. >>>> I think I've got 4-6 ports trees that I use with poudriere. >>> Which I have 2 currently - one which is 'HEAD' and the other which is my >>> 'pkg_*' tools tree (up to date - mostly). >>>> The repository that gets updated by a poudriere run is named >>>> with a combination of the jail name, the ports tree name, and the set >>>> name (-z option). The latter can be use to select an alternate >>>> make.conf to set different port options. >>> Yes, however 3.1.x 'updated' the repo from pkg_* to something like pkgng >>> - it was completely f**ked though... basically had to erase everything, >>> downgrade and reinstall everything to get it back to a 'will build both >>> trees' state. >>> >>>>> Question is why would it be needed? Surely the tree is the tree in the >>>>> jail and has nothing to do with the host? or is it not a case of >>>>> everything is done in the jail, just the actual building is and >>>>> therefore I need new build servers for the NG tree.. Which basically >>>>> means I should just decide to fork or erase the whole system because I >>>>> can't "NG" right now and I can't actually continue to build in parallel >>>>> because of this breakage? >>>> Only the actual building is done in jails. When poudriere first starts >>>> up, it looks at the list of ports that you want to build and then uses >>>> the Makefiles for each of those ports to determine the dependencies of >>>> each and the proper build order. >>> That doesn't make sense... the host has for months been well behind >>> both my tree and the ng trees... all the versions would be way out of >>> whack and even some not existing (ruby comes to mind.) >> As I mentioned previously, poudriere is just a shell script so it isn't >> tightly bound to the version of the ports tree. What caused the problem >> that you are seeing now is that the ports framework changed in a way >> that is not compatible with really old versions of poudriere. Basically >> the output of "make -V BUILD_DEPENDS" changed from >> blah:/path/to/ports/category/port to blah:category/port, and the old >> version of poudriere can't cope with that. > > I find that 'odd' at first parse... though will take your word for it > and check the code... maybe make my own version of poudriere that can > handle both... With a ports tree from late last year: %cd /usr/ports/lang/gcc make -V BUILD_DEPENDS /usr/local/bin/as:/usr/ports/devel/binutils gmake:/usr/ports/devel/gmake libiconv>=1.14_9:/usr/ports/converters/libiconv /usr/local/share/java/ecj-4.5.jar:/usr/ports/lang/gcc-ecj45 zip:/usr/ports/archivers/zip /usr/local/bin/as:/usr/ports/devel/binutils perl5>=5.20<5.21:/usr/ports/lang/perl5.20 >From a recent ports tree: %make -V BUILD_DEPENDS /usr/local/bin/as:devel/binutils gmake:devel/gmake libiconv>=1.14_9:converters/libiconv makeinfo:print/texinfo /usr/local/share/java/ecj-4.5.jar:lang/gcc-ecj45 zip:archivers/zip /usr/local/bin/as:devel/binutils perl5>=5.20<5.21:lang/perl5.20 It turns out that there are more moving parts in poudriere that I expected. Patching your copy of poudriere looks like your best bet. That will also allow you to build old-style packages with recent versions of the ports tree. Look for changes between version 3.1.8 and 3.1.9 here: <https://github.com/freebsd/poudriere/>; >> >> >>>> The ports tree used by each build jail is not related to any ports tree >>>> on the host (unless you do something like "poudriere ports -c -F -f none >>>> -M /usr/ports -p systemports", see >>>> <https://fossil.etoilebsd.net/poudriere/doc/trunk/doc/use_system_ports_tree.wiki>). >>>> It's possible to run poudriere without having /usr/ports installed on >>>> the host. >>> Will check this - if it mounts the local copy of the tree this would >>> probably fix it. >>>> If you think this is a security risk, you could run poudriere in a VM. >>> They already are in VMs.. but if poudriere make modifications to the OS >>> then it is a security issue. If it modifies/builds packages that's fine. >>> >>> ... let me expand on that ... anything that modifies something in the >>> base OS unless specifically designed and approved to interact with the >>> OS (eg puppet) then as far as I am concerned (and my employer) it's a >>> security risk. Can't have things willy nilly changing the OS, it will >>> eventually break stuff and that could cause/lead to production >>> outages... it's just not done. >> I don't think it's changing anything in the base OS. > > Well bapt swears that it wasn't freebsd-update that screwed the OS, so > must be pourdiere ... don't know both changes happened at the same time, > and it screwed my migration plans to the point I've told $employer to > forget looking at FreeBSD at all for internal systems or embedded > systems and stick with the current OS of RHEL/CentOS and please assign > people to take over my machines and change them to match the corp > standard... So far they have refused to switch them (lack of resources) > and after a long discussion with a person in OPs that also has an > @freebsd.org address it was suggested I just fork the OS and ports > tree... However, after news that i might not have as long as I hoped, I > think this would be folly and a waste of my remaining days. Funny thing that you should mention freebsd-update ... in another thread a couple people mentioned getting the wrong version of ifconfig after running freebsd-update. >> It sounds to me >> like the problem is confined to upgrading old-style pkg repositories to >> the NG when you try to update some of the packages contained in them, >> which is not what you want in this case. > > Yeah that happened as well. >> >> You should be able to install a newer version of poudriere alongside the >> old one. > That occurred to me, however unless its changed a lot it'll really not > work unless chrooted elsewhere because of shared scripts... or > rebuilding it to handle the shared stuff in different places (assuming > not already able to cope) > >> Just call it poudriere-ng or something. Just be sure not to >> run it on one of your old pkg repositories. > That's not going to be an issue... will probably just have to put in > new VMs... just about got everything off 9.1 and pretty sure nothing is > on 9.0 now so maybe re-purpose them to be NG only hosts. >> You might even be able to >> edit the script to bail out if it detects and old-style pkg repository >> as an anti-foot-shooting measure. > > Think it would need to be physically separated because I know if bapt > was not lying about freebsd-update making the changes then it must have > been poudriere that changed the OS... so not worth taking the risk. > > For the time being, just disable to entire NG repo keep my fork going > with the working (mostly) up to date ports tree for pkg_* tools and well > if the worst comes to the worst it'll fall to someone else in my absence > and they can deal with switching to RHEL/CentOS because they'd never > work out how the SORBS FreeBSD build system is put together. > > Thanks for the replies though Don, all the best. >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?407950.49499.bm>