Date: Thu, 4 Feb 2021 20:37:49 +0100 From: Henrik Rosenke <rosenke@dssgmbh.de> To: freebsd-ports@freebsd.org Subject: poudriere merging multiple ports trees Message-ID: <71f76ced-ad0b-36b2-9398-016d9c811396@dssgmbh.de> In-Reply-To: <3f87ebbe-7ecc-9f52-e556-eeb939e7acd2@dssgmbh.de> References: <3ea16730-84a1-52ce-2251-bdd808fe5c52@pinyon.org> <4990d009-1962-62c2-3f4e-4f62fd03e26d@madpilot.net> <20210125141051.qheedhp76d2ha6zy@ivaldir.net> <3f2d08b2-2959-dd3e-89e5-895437e44040@quip.cz> <20210125152825.mnd4qhi4k23dy6dw@ivaldir.net> <0e79482c-fd7c-87a2-0052-5d7eeffd3099@dssgmbh.de> <a17ab317-b023-3b35-52a7-4952786742dc@dssgmbh.de> <b0fa1609-2a58-d68e-57c7-337f5101379b@quip.cz> <a6886f94-ed37-8241-3884-5dc56a149835@dssgmbh.de> <e66b47f4-6fa2-70cf-830a-31b717e9ce56@quip.cz> <1f0d7c6c-f97e-32a1-6639-512c804431df@dssgmbh.de> <3f87ebbe-7ecc-9f52-e556-eeb939e7acd2@dssgmbh.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Greetings,
this patch had a few errors that are fixed with this patch, checking for
NOT_MOVED and dont produce an error if its not there and also dont
delete packages that are in NOT_MOVED to build them again.
This Patch targets poudriere-devel 3.3.99.20200326_2
--- common.sh.1st 2021-01-27 15:44:15.000000000 +0100
+++ common.sh 2021-02-04 20:30:42.061211000 +0100
@@ -5040,7 +5040,10 @@
fi
if shash_get origin-moved "${origin}" new_origin; then
- if [ "${new_origin%% *}" = "EXPIRED" ]; then
+ # check also against NOT_MOVED from overlays so we dont
delete this package if its wanted
+ if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS#
}/NOT_MOVED ] && (grep -q "${origin}"
${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED); then
+ msg_debug "${origin} is found in
${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED so dont delete this
package."
+ elif [ "${new_origin%% *}" = "EXPIRED" ]; then
msg "Deleting ${pkg##*/}:
${COLOR_PORT}${origin}${COLOR_RESET} ${new_origin#EXPIRED }"
else
msg "Deleting ${pkg##*/}:
${COLOR_PORT}${origin}${COLOR_RESET} moved to
${COLOR_PORT}${new_origin}${COLOR_RESET}"
@@ -6815,8 +6818,11 @@
continue
fi
origin_listed="${origin}"
- if shash_get origin-moved "${origin}" new_origin; then
- if [ "${new_origin%% *}" = "EXPIRED" ]; then
+ # check against the overlay list, skip port if its in there
+ if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS#
}/NOT_MOVED ] && (grep -q "${origin}"
${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED); then
+ msg_debug "${origin} found in
${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED, skipped checking
MOVED File of master portstree"
+ elif shash_get origin-moved "${origin}" new_origin; then
+ if [ "${new_origin%% *}" = "EXPIRED" ] ; then
msg_error "MOVED: ${origin} ${new_origin}"
set_dep_fatal_error
continue
@@ -7085,6 +7091,10 @@
fi
[ -f ${MASTERMNT}${PORTSDIR}/MOVED ] || return 0
msg "Loading MOVED for ${MASTERMNT}${PORTSDIR}"
+ # Respect overlays, ${OVERLAYS} has a leading whitepace
+ if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED ]; then
+ msg "Loading NOT_MOVED from overlaydir:
${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }"
+ fi
bset status "loading_moved:"
awk -f ${AWKPREFIX}/parse_MOVED.awk \
${MASTERMNT}${PORTSDIR}/MOVED | \
Kind regards,
Henrik Rosenke
Am 29.01.21 um 17:38 schrieb Henrik Rosenke:
> Greetings,
>
> i created a simple patch for this, this may not be the best solution
> but my tests seems to be successful to get the desired result:
>
> Without NOT_MOVED:
>
> =>> Debug: Reading /usr/local/etc/poudriere.conf
> [00:00:00] Creating the reference jail... done
> [00:00:02] Mounting system devices for j12p64-default-server
> [00:00:02] Mounting ports/packages/distfiles
> [00:00:02] Using packages from previously failed build:
> /home/poudriere/data/packages/j12p64-default-server/.building
> [00:00:02] Mounting ccache from: /home/poudriere/ccache
> [00:00:02] Mounting packages from:
> /home/poudriere/data/packages/j12p64-default-server
> [00:00:02] Copying /var/db/ports from:
> /usr/local/etc/poudriere.d/j12p64-options
> [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
> [00:00:02] Appending to make.conf:
> /usr/local/etc/poudriere.d/server-make.conf
> [00:00:02] Appending to make.conf:
> /usr/local/etc/poudriere.d/default-make.conf
> [00:00:02] Appending to make.conf:
> /usr/local/etc/poudriere.d/j12p64-make.conf
> [00:00:02] Appending to make.conf:
> /usr/local/etc/poudriere.d/default-server-make.conf
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): comms/chan_capi
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-kmod
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-utils
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): sysutils/env4801
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): security/ipsec-tools
> /etc/resolv.conf ->
> /home/poudriere/data/.m/j12p64-default-server/ref/etc/resolv.conf
> [00:00:02] Starting jail j12p64-default-server
> [00:00:02] Logs:
> /home/poudriere/data/logs/bulk/j12p64-default-server/2021-01-29_11h14m59s
> [00:00:02] WWW:
> http://dsspkg.incore/poudriere//build.html?mastername=j12p64-default-server&build=2021-01-29_11h14m59s
> [00:00:02] Loading MOVED for
> /home/poudriere/data/.m/j12p64-default-server/ref/usr/ports
> [00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS
> [00:00:03] Gathering ports metadata
> [00:00:03] Error: MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has
> expired: Ancient and unsupported release
> [00:00:03] Error: Fatal errors encountered gathering initial ports
> metadata
> [00:00:03] Cleaning up
> [00:00:03] Unmounting file systems
>
> With NOT_MOVED:
>
> === root@dsspkg (pts/1) /home/poudriere/ports/dss_overlay 35 -> cat
> NOT_MOVED
> sysutils/sge62
>
> === root@dsspkg (pts/1) /home/poudriere/ports/dss_overlay 97 ->
> poudriere bulk -vv -O dss_overlay -j j12p64 -z server -p default
> sysutils/sge62
> =>> Debug: Reading /usr/local/etc/poudriere.conf
> [00:00:00] Creating the reference jail... done
> [00:00:02] Mounting system devices for j12p64-default-server
> [00:00:02] Mounting ports/packages/distfiles
> [00:00:02] Using packages from previously failed build:
> /home/poudriere/data/packages/j12p64-default-server/.building
> [00:00:02] Mounting ccache from: /home/poudriere/ccache
> [00:00:02] Mounting packages from:
> /home/poudriere/data/packages/j12p64-default-server
> [00:00:02] Copying /var/db/ports from:
> /usr/local/etc/poudriere.d/j12p64-options
> [00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
> [00:00:02] Appending to make.conf:
> /usr/local/etc/poudriere.d/server-make.conf
> [00:00:02] Appending to make.conf:
> /usr/local/etc/poudriere.d/default-make.conf
> [00:00:02] Appending to make.conf:
> /usr/local/etc/poudriere.d/j12p64-make.conf
> [00:00:02] Appending to make.conf:
> /usr/local/etc/poudriere.d/default-server-make.conf
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): comms/chan_capi
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-kmod
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-utils
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): sysutils/env4801
> [00:00:02] Warning: Blacklisting (from
> /usr/local/etc/poudriere.d/j12p64-blacklist): security/ipsec-tools
> /etc/resolv.conf ->
> /home/poudriere/data/.m/j12p64-default-server/ref/etc/resolv.conf
> [00:00:02] Starting jail j12p64-default-server
> [00:00:03] Logs:
> /home/poudriere/data/logs/bulk/j12p64-default-server/2021-01-29_17h28m52s
> [00:00:03] WWW:
> http://dsspkg.incore/poudriere//build.html?mastername=j12p64-default-server&build=2021-01-29_17h28m52s
> [00:00:03] Loading MOVED for
> /home/poudriere/data/.m/j12p64-default-server/ref/usr/ports
> [00:00:03] Loading NOT_MOVED from overlaydir:
> /home/poudriere/data/.m/j12p64-default-server/ref/overlays/dss_overlay
> [00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS
> [00:00:03] Gathering ports metadata
> [00:00:03] Debug: sysutils/sge62 found in
> /home/poudriere/data/.m/j12p64-default-server/ref/overlays/dss_overlay/NOT_MOVED,
> skipped checking MOVED File of master portstree
> [00:00:03] Debug: queueing sysutils/sge62 into gatherqueue (rdep=listed)
> [00:00:03] Debug: Processing gatherqueue
> [00:00:03] Warning: (sysutils/sge62): Debug: gather_port_vars_port
> (sysutils/sge62): LOOKUP
> [00:00:03] Warning: (sysutils/sge62): Debug: deps_fetch_vars:
> discovered sysutils/sge62 is sge-6.2.2.1_6
> [00:00:03] Warning: (sysutils/sge62): Debug: WILL BUILD sysutils/sge62
>
> Patch leading to this:
>
> --- /usr/local/share/poudriere/common.sh.1st 2021-01-27
> 15:44:15.000000000 +0100
> +++ /usr/local/share/poudriere/common.sh 2021-01-29
> 17:27:14.810441000 +0100
> @@ -6815,8 +6815,11 @@
> continue
> fi
> origin_listed="${origin}"
> - if shash_get origin-moved "${origin}" new_origin; then
> - if [ "${new_origin%% *}" = "EXPIRED" ]; then
> + # check against the overlay list, skip port if its in
> there
> + if (grep -q "${origin}"
> ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED); then
> + msg_debug "${origin} found in
> ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED, skipped checking
> MOVED File of master portstree"
> + elif shash_get origin-moved "${origin}" new_origin; then
> + if [ "${new_origin%% *}" = "EXPIRED" ] ; then
> msg_error "MOVED: ${origin}
> ${new_origin}"
> set_dep_fatal_error
> continue
> @@ -7085,6 +7088,10 @@
> fi
> [ -f ${MASTERMNT}${PORTSDIR}/MOVED ] || return 0
> msg "Loading MOVED for ${MASTERMNT}${PORTSDIR}"
> + # Respect overlays, ${OVERLAYS} has a leading whitepace
> + if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED ];
> then
> + msg "Loading NOT_MOVED from overlaydir:
> ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }"
> + fi
> bset status "loading_moved:"
> awk -f ${AWKPREFIX}/parse_MOVED.awk \
> ${MASTERMNT}${PORTSDIR}/MOVED | \
>
> Maybe someone else could test this, seems to work for me.
>
>
> Kind regards,
>
> Henrik Rosenke
>
>
> Am 28.01.21 um 21:56 schrieb Henrik Rosenke:
>>
>> Am 28.01.21 um 21:14 schrieb Miroslav Lachman:
>>> On 28/01/2021 17:47, Henrik Rosenke wrote:
>>>> I tried this with a patched MOVED file, copied from the master
>>>> portstree, deleted the entry and tried to build but this File seems
>>>> to be ignored. I dont really understand where the MOVED file is
>>>> handled, it seems to be bsd.port.subdir.mk but i am not sure.
>>>
>>> MOVED is handled by bsd.port.subdir.mk but overlays do not account
>>> with it. Overlays are handled on more places
>>
>> i thought this too but it seems poudriere is canceled before this, i
>> even tried to move the bsd.port.subdir.mk and commented out the
>> function load_moved in the common.sh of poudriere and can start a
>> build. Correct me if iam wrong but it seems that the common.sh from
>> poudriere needs to be patched to handle this rather than the Mk files
>> itself. I added set -x in the common.sh to get a better understanding
>> what is happening and it seems that shash_set sucks in the MOVED file
>> and this is checked later against the ports we want to build:
>>
>> + shash_set origin-moved sysutils/sge62 'EXPIRED 2013-06-05 Has
>> expired: Ancient and unsupported release'
>>
>> later:
>>
>> + originspec_decode sysutils/sge62 origin '' flavor
>> + local -
>> + set +x -f
>> + [ -n '' ]
>> + origin_listed=sysutils/sge62
>> + shash_get origin-moved sysutils/sge62 new_origin
>> + local -
>> + set +x
>> + [ EXPIRED '=' EXPIRED ]
>> + msg_error 'MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has expired:
>> Ancient and unsupported release'
>> + local -
>> + set +x
>> [00:00:07] Error: MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has
>> expired: Ancient and unsupported release
>>
>> Kind regards,
>>
>> Henrik Rosenke
>>
>>>
>>> Scripts/do-depends.sh
>>> Scripts/depends-list.sh
>>>
>>> Next is bsd.port.mk:
>>>
>>> .if defined(USE_LOCAL_MK)
>>> .include "${PORTSDIR}/Mk/bsd.local.mk"
>>> .endif
>>> .for odir in ${OVERLAYS}
>>> .sinclude "${odir}/Mk/bsd.overlay.mk"
>>> .endfor
>>>
>>> But there is no bsd.overlay.mk and I don't know what should be in it
>>> (file in the overlay ports tree)
>>>
>>> The AWK code in bsd.port.subdir.mk can be changed to use MOVED from
>>> overlay if it exists insead of the default one.
>>>
>>>> I think the best way to handle this would be a NOT_MOVED file in
>>>> the overlay portstree with the ports to ignore from the MOVED file
>>>> with entries like "sysutils/sge62".
>>>
>>> This can be possible too. NOT_MOVED can be used as additional INDEXFILE
>>>
>>> Or maybe start with PORTSEARCH_MOVED=0 (not tried yet)
>>>
>>> Kind regards
>>> Miroslav Lachman
> _______________________________________________
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?71f76ced-ad0b-36b2-9398-016d9c811396>