From owner-freebsd-ports@freebsd.org Thu Feb 4 19:38:02 2021 Return-Path: Delivered-To: freebsd-ports@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3247952D263 for ; Thu, 4 Feb 2021 19:38:02 +0000 (UTC) (envelope-from rosenke@dssgmbh.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DWpk12PWwz4pjD for ; Thu, 4 Feb 2021 19:38:00 +0000 (UTC) (envelope-from rosenke@dssgmbh.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 1346B3AD7F for ; Thu, 4 Feb 2021 20:37:52 +0100 (CET) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id dMQY9yc3GEu7 for ; Thu, 4 Feb 2021 20:37:50 +0100 (CET) Received: from secmail.incore.de (inetdssm.dmz [10.11.0.4]) by dss.incore.de (Postfix) with ESMTP id 12C803ACDB for ; Thu, 4 Feb 2021 20:37:50 +0100 (CET) Received: from [192.168.10.33] (dslb-002-204-037-022.002.204.pools.vodafone-ip.de [2.204.37.22]) by secmail.incore.de (Postfix) with ESMTPS id 033B213C1C for ; Thu, 4 Feb 2021 20:37:49 +0100 (CET) Subject: poudriere merging multiple ports trees To: freebsd-ports@freebsd.org References: <3ea16730-84a1-52ce-2251-bdd808fe5c52@pinyon.org> <4990d009-1962-62c2-3f4e-4f62fd03e26d@madpilot.net> <20210125141051.qheedhp76d2ha6zy@ivaldir.net> <3f2d08b2-2959-dd3e-89e5-895437e44040@quip.cz> <20210125152825.mnd4qhi4k23dy6dw@ivaldir.net> <0e79482c-fd7c-87a2-0052-5d7eeffd3099@dssgmbh.de> <1f0d7c6c-f97e-32a1-6639-512c804431df@dssgmbh.de> <3f87ebbe-7ecc-9f52-e556-eeb939e7acd2@dssgmbh.de> From: Henrik Rosenke Message-ID: <71f76ced-ad0b-36b2-9398-016d9c811396@dssgmbh.de> Date: Thu, 4 Feb 2021 20:37:49 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: <3f87ebbe-7ecc-9f52-e556-eeb939e7acd2@dssgmbh.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4DWpk12PWwz4pjD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rosenke@dssgmbh.de designates 195.145.1.138 as permitted sender) smtp.mailfrom=rosenke@dssgmbh.de X-Spamd-Result: default: False [-3.30 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[195.145.1.138:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.145.1.138]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ports@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[195.145.1.138:from:127.0.2.255]; DMARC_NA(0.00)[dssgmbh.de]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3320, ipnet:195.145.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-ports]; RECEIVED_SPAMHAUS_PBL(0.00)[2.204.37.22:received] X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2021 19:38:02 -0000 Greetings, this patch had a few errors that are fixed with this patch, checking for NOT_MOVED and dont produce an error if its not there and also dont delete packages that are in NOT_MOVED to build them again. This Patch targets poudriere-devel 3.3.99.20200326_2 --- common.sh.1st       2021-01-27 15:44:15.000000000 +0100 +++ common.sh   2021-02-04 20:30:42.061211000 +0100 @@ -5040,7 +5040,10 @@         fi         if shash_get origin-moved "${origin}" new_origin; then -               if [ "${new_origin%% *}" = "EXPIRED" ]; then +               # check also against NOT_MOVED from overlays so we dont delete this package if its wanted +               if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED ] && (grep -q "${origin}" ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED); then +                       msg_debug "${origin} is found in ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED so dont delete this package." +               elif [ "${new_origin%% *}" = "EXPIRED" ]; then                         msg "Deleting ${pkg##*/}: ${COLOR_PORT}${origin}${COLOR_RESET} ${new_origin#EXPIRED }"                 else                         msg "Deleting ${pkg##*/}: ${COLOR_PORT}${origin}${COLOR_RESET} moved to ${COLOR_PORT}${new_origin}${COLOR_RESET}" @@ -6815,8 +6818,11 @@                         continue                 fi                 origin_listed="${origin}" -               if shash_get origin-moved "${origin}" new_origin; then -                       if [ "${new_origin%% *}" = "EXPIRED" ]; then +               # check against the overlay list, skip port if its in there +               if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED ] && (grep -q "${origin}" ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED); then +                       msg_debug "${origin} found in ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED, skipped checking MOVED File of master portstree" +               elif shash_get origin-moved "${origin}" new_origin; then +                       if [ "${new_origin%% *}" = "EXPIRED" ] ; then                                 msg_error "MOVED: ${origin} ${new_origin}"                                 set_dep_fatal_error                                 continue @@ -7085,6 +7091,10 @@         fi         [ -f ${MASTERMNT}${PORTSDIR}/MOVED ] || return 0         msg "Loading MOVED for ${MASTERMNT}${PORTSDIR}" +       # Respect overlays, ${OVERLAYS} has a leading whitepace +       if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED ]; then +               msg "Loading NOT_MOVED from overlaydir: ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }" +       fi         bset status "loading_moved:"         awk -f ${AWKPREFIX}/parse_MOVED.awk \             ${MASTERMNT}${PORTSDIR}/MOVED | \ Kind regards, Henrik Rosenke Am 29.01.21 um 17:38 schrieb Henrik Rosenke: > Greetings, > > i created a simple patch for this, this may not be the best solution > but my tests seems to be successful to get the desired result: > > Without NOT_MOVED: > > =>> Debug: Reading /usr/local/etc/poudriere.conf > [00:00:00] Creating the reference jail... done > [00:00:02] Mounting system devices for j12p64-default-server > [00:00:02] Mounting ports/packages/distfiles > [00:00:02] Using packages from previously failed build: > /home/poudriere/data/packages/j12p64-default-server/.building > [00:00:02] Mounting ccache from: /home/poudriere/ccache > [00:00:02] Mounting packages from: > /home/poudriere/data/packages/j12p64-default-server > [00:00:02] Copying /var/db/ports from: > /usr/local/etc/poudriere.d/j12p64-options > [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf > [00:00:02] Appending to make.conf: > /usr/local/etc/poudriere.d/server-make.conf > [00:00:02] Appending to make.conf: > /usr/local/etc/poudriere.d/default-make.conf > [00:00:02] Appending to make.conf: > /usr/local/etc/poudriere.d/j12p64-make.conf > [00:00:02] Appending to make.conf: > /usr/local/etc/poudriere.d/default-server-make.conf > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): comms/chan_capi > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-kmod > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-utils > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): sysutils/env4801 > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): security/ipsec-tools > /etc/resolv.conf -> > /home/poudriere/data/.m/j12p64-default-server/ref/etc/resolv.conf > [00:00:02] Starting jail j12p64-default-server > [00:00:02] Logs: > /home/poudriere/data/logs/bulk/j12p64-default-server/2021-01-29_11h14m59s > [00:00:02] WWW: > http://dsspkg.incore/poudriere//build.html?mastername=j12p64-default-server&build=2021-01-29_11h14m59s > [00:00:02] Loading MOVED for > /home/poudriere/data/.m/j12p64-default-server/ref/usr/ports > [00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS > [00:00:03] Gathering ports metadata > [00:00:03] Error: MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has > expired: Ancient and unsupported release > [00:00:03] Error: Fatal errors encountered gathering initial ports > metadata > [00:00:03] Cleaning up > [00:00:03] Unmounting file systems > > With NOT_MOVED: > > === root@dsspkg (pts/1) /home/poudriere/ports/dss_overlay 35 -> cat > NOT_MOVED > sysutils/sge62 > > === root@dsspkg (pts/1) /home/poudriere/ports/dss_overlay 97 -> > poudriere bulk -vv -O dss_overlay -j j12p64 -z server -p default > sysutils/sge62 > =>> Debug: Reading /usr/local/etc/poudriere.conf > [00:00:00] Creating the reference jail... done > [00:00:02] Mounting system devices for j12p64-default-server > [00:00:02] Mounting ports/packages/distfiles > [00:00:02] Using packages from previously failed build: > /home/poudriere/data/packages/j12p64-default-server/.building > [00:00:02] Mounting ccache from: /home/poudriere/ccache > [00:00:02] Mounting packages from: > /home/poudriere/data/packages/j12p64-default-server > [00:00:02] Copying /var/db/ports from: > /usr/local/etc/poudriere.d/j12p64-options > [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf > [00:00:02] Appending to make.conf: > /usr/local/etc/poudriere.d/server-make.conf > [00:00:02] Appending to make.conf: > /usr/local/etc/poudriere.d/default-make.conf > [00:00:02] Appending to make.conf: > /usr/local/etc/poudriere.d/j12p64-make.conf > [00:00:02] Appending to make.conf: > /usr/local/etc/poudriere.d/default-server-make.conf > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): comms/chan_capi > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-kmod > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-utils > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): sysutils/env4801 > [00:00:02] Warning: Blacklisting (from > /usr/local/etc/poudriere.d/j12p64-blacklist): security/ipsec-tools > /etc/resolv.conf -> > /home/poudriere/data/.m/j12p64-default-server/ref/etc/resolv.conf > [00:00:02] Starting jail j12p64-default-server > [00:00:03] Logs: > /home/poudriere/data/logs/bulk/j12p64-default-server/2021-01-29_17h28m52s > [00:00:03] WWW: > http://dsspkg.incore/poudriere//build.html?mastername=j12p64-default-server&build=2021-01-29_17h28m52s > [00:00:03] Loading MOVED for > /home/poudriere/data/.m/j12p64-default-server/ref/usr/ports > [00:00:03] Loading NOT_MOVED from overlaydir: > /home/poudriere/data/.m/j12p64-default-server/ref/overlays/dss_overlay > [00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS > [00:00:03] Gathering ports metadata > [00:00:03] Debug: sysutils/sge62 found in > /home/poudriere/data/.m/j12p64-default-server/ref/overlays/dss_overlay/NOT_MOVED, > skipped checking MOVED File of master portstree > [00:00:03] Debug: queueing sysutils/sge62 into gatherqueue (rdep=listed) > [00:00:03] Debug: Processing gatherqueue > [00:00:03] Warning: (sysutils/sge62): Debug: gather_port_vars_port > (sysutils/sge62): LOOKUP > [00:00:03] Warning: (sysutils/sge62): Debug: deps_fetch_vars: > discovered sysutils/sge62 is sge-6.2.2.1_6 > [00:00:03] Warning: (sysutils/sge62): Debug: WILL BUILD sysutils/sge62 > > Patch leading to this: > > --- /usr/local/share/poudriere/common.sh.1st    2021-01-27 > 15:44:15.000000000 +0100 > +++ /usr/local/share/poudriere/common.sh        2021-01-29 > 17:27:14.810441000 +0100 > @@ -6815,8 +6815,11 @@ >                         continue >                 fi >                 origin_listed="${origin}" > -               if shash_get origin-moved "${origin}" new_origin; then > -                       if [ "${new_origin%% *}" = "EXPIRED" ]; then > +               # check against the overlay list, skip port if its in > there > +               if (grep -q "${origin}" > ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED); then > +                       msg_debug "${origin} found in > ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED, skipped checking > MOVED File of master portstree" > +               elif shash_get origin-moved "${origin}" new_origin; then > +                       if [ "${new_origin%% *}" = "EXPIRED" ] ; then >                                 msg_error "MOVED: ${origin} > ${new_origin}" >                                 set_dep_fatal_error >                                 continue > @@ -7085,6 +7088,10 @@ >         fi >         [ -f ${MASTERMNT}${PORTSDIR}/MOVED ] || return 0 >         msg "Loading MOVED for ${MASTERMNT}${PORTSDIR}" > +       # Respect overlays, ${OVERLAYS} has a leading whitepace > +       if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED ]; > then > +               msg "Loading NOT_MOVED from overlaydir: > ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }" > +       fi >         bset status "loading_moved:" >         awk -f ${AWKPREFIX}/parse_MOVED.awk \ >             ${MASTERMNT}${PORTSDIR}/MOVED | \ > > Maybe someone else could test this, seems to work for me. > > > Kind regards, > > Henrik Rosenke > > > Am 28.01.21 um 21:56 schrieb Henrik Rosenke: >> >> Am 28.01.21 um 21:14 schrieb Miroslav Lachman: >>> On 28/01/2021 17:47, Henrik Rosenke wrote: >>>> I tried this with a patched MOVED file, copied from the master >>>> portstree, deleted the entry and tried to build but this File seems >>>> to be ignored. I dont really understand where the MOVED file is >>>> handled, it seems to be bsd.port.subdir.mk but i am not sure. >>> >>> MOVED is handled by bsd.port.subdir.mk but overlays do not account >>> with it. Overlays are handled on more places >> >> i thought this too but it seems poudriere is canceled before this, i >> even tried to move the bsd.port.subdir.mk and commented out the >> function load_moved in the common.sh of poudriere and can start a >> build. Correct me if iam wrong but it seems that the common.sh from >> poudriere needs to be patched to handle this rather than the Mk files >> itself. I added set -x in the common.sh to get a better understanding >> what is happening and it seems that shash_set sucks in the MOVED file >> and this is checked later against the ports we want to build: >> >> + shash_set origin-moved sysutils/sge62 'EXPIRED 2013-06-05 Has >> expired: Ancient and unsupported release' >> >> later: >> >> + originspec_decode sysutils/sge62 origin '' flavor >> + local - >> + set +x -f >> + [ -n '' ] >> + origin_listed=sysutils/sge62 >> + shash_get origin-moved sysutils/sge62 new_origin >> + local - >> + set +x >> + [ EXPIRED '=' EXPIRED ] >> + msg_error 'MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has expired: >> Ancient and unsupported release' >> + local - >> + set +x >> [00:00:07] Error: MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has >> expired: Ancient and unsupported release >> >> Kind regards, >> >> Henrik Rosenke >> >>> >>> Scripts/do-depends.sh >>> Scripts/depends-list.sh >>> >>> Next is bsd.port.mk: >>> >>> .if defined(USE_LOCAL_MK) >>> .include "${PORTSDIR}/Mk/bsd.local.mk" >>> .endif >>> .for odir in ${OVERLAYS} >>> .sinclude "${odir}/Mk/bsd.overlay.mk" >>> .endfor >>> >>> But there is no bsd.overlay.mk and I don't know what should be in it >>> (file in the overlay ports tree) >>> >>> The AWK code in bsd.port.subdir.mk can be changed to use MOVED from >>> overlay if it exists insead of the default one. >>> >>>> I think the best way to handle this would be a NOT_MOVED file in >>>> the overlay portstree with the ports to ignore from the MOVED file >>>> with entries like "sysutils/sge62". >>> >>> This can be possible too. NOT_MOVED can be used as additional INDEXFILE >>> >>> Or maybe start with PORTSEARCH_MOVED=0 (not tried yet) >>> >>> Kind regards >>> Miroslav Lachman > _______________________________________________ > freebsd-ports@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org"