From owner-freebsd-ports@freebsd.org Fri Jan 29 16:38:36 2021 Return-Path: Delivered-To: freebsd-ports@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4B8B74F15E8 for ; Fri, 29 Jan 2021 16:38:36 +0000 (UTC) (envelope-from rosenke@dssgmbh.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DS31l3mR5z3MSC for ; Fri, 29 Jan 2021 16:38:35 +0000 (UTC) (envelope-from rosenke@dssgmbh.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 6B92A3ADF8; Fri, 29 Jan 2021 17:38:33 +0100 (CET) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id HdoCTldd4PsX; Fri, 29 Jan 2021 17:38:31 +0100 (CET) Received: from secmail.incore.de (inetdssm.dmz [10.11.0.4]) by dss.incore.de (Postfix) with ESMTP id 733673ACF0; Fri, 29 Jan 2021 17:38:31 +0100 (CET) Received: from [192.168.10.33] (dslb-002-204-037-022.002.204.pools.vodafone-ip.de [2.204.37.22]) by secmail.incore.de (Postfix) with ESMTPS id 6276713C04; Fri, 29 Jan 2021 17:38:31 +0100 (CET) Subject: Re: poudriere merging multiple ports trees From: Henrik Rosenke To: Miroslav Lachman <000.fbsd@quip.cz>, FreeBSD References: <3ea16730-84a1-52ce-2251-bdd808fe5c52@pinyon.org> <4990d009-1962-62c2-3f4e-4f62fd03e26d@madpilot.net> <20210125141051.qheedhp76d2ha6zy@ivaldir.net> <3f2d08b2-2959-dd3e-89e5-895437e44040@quip.cz> <20210125152825.mnd4qhi4k23dy6dw@ivaldir.net> <0e79482c-fd7c-87a2-0052-5d7eeffd3099@dssgmbh.de> <1f0d7c6c-f97e-32a1-6639-512c804431df@dssgmbh.de> Message-ID: <3f87ebbe-7ecc-9f52-e556-eeb939e7acd2@dssgmbh.de> Date: Fri, 29 Jan 2021 17:38:31 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: <1f0d7c6c-f97e-32a1-6639-512c804431df@dssgmbh.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4DS31l3mR5z3MSC X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rosenke@dssgmbh.de designates 195.145.1.138 as permitted sender) smtp.mailfrom=rosenke@dssgmbh.de X-Spamd-Result: default: False [-1.30 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[195.145.1.138:from]; RCVD_COUNT_FIVE(0.00)[5]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.145.1.138:c]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[dssgmbh.de]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[195.145.1.138:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3320, ipnet:195.145.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-ports]; RECEIVED_SPAMHAUS_PBL(0.00)[2.204.37.22:received] X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2021 16:38:36 -0000 Greetings, i created a simple patch for this, this may not be the best solution but my tests seems to be successful to get the desired result: Without NOT_MOVED: =>> Debug: Reading /usr/local/etc/poudriere.conf [00:00:00] Creating the reference jail... done [00:00:02] Mounting system devices for j12p64-default-server [00:00:02] Mounting ports/packages/distfiles [00:00:02] Using packages from previously failed build: /home/poudriere/data/packages/j12p64-default-server/.building [00:00:02] Mounting ccache from: /home/poudriere/ccache [00:00:02] Mounting packages from: /home/poudriere/data/packages/j12p64-default-server [00:00:02] Copying /var/db/ports from: /usr/local/etc/poudriere.d/j12p64-options [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/server-make.conf [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/default-make.conf [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/j12p64-make.conf [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/default-server-make.conf [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): comms/chan_capi [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-kmod [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-utils [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): sysutils/env4801 [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): security/ipsec-tools /etc/resolv.conf -> /home/poudriere/data/.m/j12p64-default-server/ref/etc/resolv.conf [00:00:02] Starting jail j12p64-default-server [00:00:02] Logs: /home/poudriere/data/logs/bulk/j12p64-default-server/2021-01-29_11h14m59s [00:00:02] WWW: http://dsspkg.incore/poudriere//build.html?mastername=j12p64-default-server&build=2021-01-29_11h14m59s [00:00:02] Loading MOVED for /home/poudriere/data/.m/j12p64-default-server/ref/usr/ports [00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS [00:00:03] Gathering ports metadata [00:00:03] Error: MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has expired: Ancient and unsupported release [00:00:03] Error: Fatal errors encountered gathering initial ports metadata [00:00:03] Cleaning up [00:00:03] Unmounting file systems With NOT_MOVED: === root@dsspkg (pts/1) /home/poudriere/ports/dss_overlay 35 -> cat NOT_MOVED sysutils/sge62 === root@dsspkg (pts/1) /home/poudriere/ports/dss_overlay 97 -> poudriere bulk -vv -O dss_overlay -j j12p64 -z server -p default sysutils/sge62 =>> Debug: Reading /usr/local/etc/poudriere.conf [00:00:00] Creating the reference jail... done [00:00:02] Mounting system devices for j12p64-default-server [00:00:02] Mounting ports/packages/distfiles [00:00:02] Using packages from previously failed build: /home/poudriere/data/packages/j12p64-default-server/.building [00:00:02] Mounting ccache from: /home/poudriere/ccache [00:00:02] Mounting packages from: /home/poudriere/data/packages/j12p64-default-server [00:00:02] Copying /var/db/ports from: /usr/local/etc/poudriere.d/j12p64-options [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/server-make.conf [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/default-make.conf [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/j12p64-make.conf [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/default-server-make.conf [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): comms/chan_capi [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-kmod [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-utils [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): sysutils/env4801 [00:00:02] Warning: Blacklisting (from /usr/local/etc/poudriere.d/j12p64-blacklist): security/ipsec-tools /etc/resolv.conf -> /home/poudriere/data/.m/j12p64-default-server/ref/etc/resolv.conf [00:00:02] Starting jail j12p64-default-server [00:00:03] Logs: /home/poudriere/data/logs/bulk/j12p64-default-server/2021-01-29_17h28m52s [00:00:03] WWW: http://dsspkg.incore/poudriere//build.html?mastername=j12p64-default-server&build=2021-01-29_17h28m52s [00:00:03] Loading MOVED for /home/poudriere/data/.m/j12p64-default-server/ref/usr/ports [00:00:03] Loading NOT_MOVED from overlaydir: /home/poudriere/data/.m/j12p64-default-server/ref/overlays/dss_overlay [00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS [00:00:03] Gathering ports metadata [00:00:03] Debug: sysutils/sge62 found in /home/poudriere/data/.m/j12p64-default-server/ref/overlays/dss_overlay/NOT_MOVED, skipped checking MOVED File of master portstree [00:00:03] Debug: queueing sysutils/sge62 into gatherqueue (rdep=listed) [00:00:03] Debug: Processing gatherqueue [00:00:03] Warning: (sysutils/sge62): Debug: gather_port_vars_port (sysutils/sge62): LOOKUP [00:00:03] Warning: (sysutils/sge62): Debug: deps_fetch_vars: discovered sysutils/sge62 is sge-6.2.2.1_6 [00:00:03] Warning: (sysutils/sge62): Debug: WILL BUILD sysutils/sge62 Patch leading to this: --- /usr/local/share/poudriere/common.sh.1st    2021-01-27 15:44:15.000000000 +0100 +++ /usr/local/share/poudriere/common.sh        2021-01-29 17:27:14.810441000 +0100 @@ -6815,8 +6815,11 @@                         continue                 fi                 origin_listed="${origin}" -               if shash_get origin-moved "${origin}" new_origin; then -                       if [ "${new_origin%% *}" = "EXPIRED" ]; then +               # check against the overlay list, skip port if its in there +               if (grep -q "${origin}" ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED); then +                       msg_debug "${origin} found in ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED, skipped checking MOVED File of master portstree" +               elif shash_get origin-moved "${origin}" new_origin; then +                       if [ "${new_origin%% *}" = "EXPIRED" ] ; then                                 msg_error "MOVED: ${origin} ${new_origin}"                                 set_dep_fatal_error                                 continue @@ -7085,6 +7088,10 @@         fi         [ -f ${MASTERMNT}${PORTSDIR}/MOVED ] || return 0         msg "Loading MOVED for ${MASTERMNT}${PORTSDIR}" +       # Respect overlays, ${OVERLAYS} has a leading whitepace +       if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED ]; then +               msg "Loading NOT_MOVED from overlaydir: ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }" +       fi         bset status "loading_moved:"         awk -f ${AWKPREFIX}/parse_MOVED.awk \             ${MASTERMNT}${PORTSDIR}/MOVED | \ Maybe someone else could test this, seems to work for me. Kind regards, Henrik Rosenke Am 28.01.21 um 21:56 schrieb Henrik Rosenke: > > Am 28.01.21 um 21:14 schrieb Miroslav Lachman: >> On 28/01/2021 17:47, Henrik Rosenke wrote: >>> I tried this with a patched MOVED file, copied from the master >>> portstree, deleted the entry and tried to build but this File seems >>> to be ignored. I dont really understand where the MOVED file is >>> handled, it seems to be bsd.port.subdir.mk but i am not sure. >> >> MOVED is handled by bsd.port.subdir.mk but overlays do not account >> with it. Overlays are handled on more places > > i thought this too but it seems poudriere is canceled before this, i > even tried to move the bsd.port.subdir.mk and commented out the > function load_moved in the common.sh of poudriere and can start a > build. Correct me if iam wrong but it seems that the common.sh from > poudriere needs to be patched to handle this rather than the Mk files > itself. I added set -x in the common.sh to get a better understanding > what is happening and it seems that shash_set sucks in the MOVED file > and this is checked later against the ports we want to build: > > + shash_set origin-moved sysutils/sge62 'EXPIRED 2013-06-05 Has > expired: Ancient and unsupported release' > > later: > > + originspec_decode sysutils/sge62 origin '' flavor > + local - > + set +x -f > + [ -n '' ] > + origin_listed=sysutils/sge62 > + shash_get origin-moved sysutils/sge62 new_origin > + local - > + set +x > + [ EXPIRED '=' EXPIRED ] > + msg_error 'MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has expired: > Ancient and unsupported release' > + local - > + set +x > [00:00:07] Error: MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has > expired: Ancient and unsupported release > > Kind regards, > > Henrik Rosenke > >> >> Scripts/do-depends.sh >> Scripts/depends-list.sh >> >> Next is bsd.port.mk: >> >> .if defined(USE_LOCAL_MK) >> .include "${PORTSDIR}/Mk/bsd.local.mk" >> .endif >> .for odir in ${OVERLAYS} >> .sinclude "${odir}/Mk/bsd.overlay.mk" >> .endfor >> >> But there is no bsd.overlay.mk and I don't know what should be in it >> (file in the overlay ports tree) >> >> The AWK code in bsd.port.subdir.mk can be changed to use MOVED from >> overlay if it exists insead of the default one. >> >>> I think the best way to handle this would be a NOT_MOVED file in the >>> overlay portstree with the ports to ignore from the MOVED file with >>> entries like "sysutils/sge62". >> >> This can be possible too. NOT_MOVED can be used as additional INDEXFILE >> >> Or maybe start with PORTSEARCH_MOVED=0 (not tried yet) >> >> Kind regards >> Miroslav Lachman