From nobody Tue Feb 7 17:56:58 2023 X-Original-To: dev-commits-ports-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PB9ns1W9Fz3nW6c for ; Tue, 7 Feb 2023 17:57:37 +0000 (UTC) (envelope-from sunpoet@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PB9ns0T91z4KB1 for ; Tue, 7 Feb 2023 17:57:37 +0000 (UTC) (envelope-from sunpoet@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675792657; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6AU79LNekmuZr2/IPrAV0BqqXI3LzY6a2LU+Mtmadcc=; b=GrMQtRNziHHRav+SdF+PeUfTMT+M518lcrLefD7RsS9W9juF9O5ixtILZy75SFwOUkR/oJ QTBozhm2gqnqW+/Grj7Af0N/ivXjIGefbnP+5nSyPf/0+biuDxzrnVgknQ+SLZ40gaBtDb A/dV1WeO44TAsJm3H/cAPS2C5IBXrHR0Cje/9f7sq40WgXb8AySEZuqwXdYnHBG4+NlRhn EuzStuWmMgmSjokkg0uFp+pSFt/T3B1dgXwc7SsY4fuRePIE5m7EZDSS6kX1s5Mdef7Z4o mjRyuthK7dEFZhmjbHr4IVGH4cD2W3JlAleY83/1/Q9Z1ZBMPuUmBhODqrY3IA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675792657; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6AU79LNekmuZr2/IPrAV0BqqXI3LzY6a2LU+Mtmadcc=; b=Xr+pT4B+VeJ/yNn6sEG82/3YHIsOx1LqylE2NW4kfDGCWv1kQk3cgGkSZGXSalg7mj/5tX FatubmQ7WKcXXbh6BQNzD9PPFDprpg4bAopB8jbgrfrE1CsZ/jmUN4O5voLW/74Xjaz83A ss0hpTVPoPW88RQk/wze5t/yw/9kYs0aoj0GipWvUNH2vH+CNEiYT01ZUnKwQgsWyQ8pHU HzeX4fBm5/A89Om2NWdhRGzlQLjhw2yEvqTb3r7vQNyiUzzHnTEbMdEJTPGSzBPu1SdVUa PYFMCEz6hUYHFYCcze+KiMVEPXc/abm2tRq8O8l2w3QHxMLWssWgijYDonxc5w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675792657; a=rsa-sha256; cv=none; b=C0sWmust/0hC75KD4Eim7s1IdtzLOrK05ZHOgxBldARzap7BROaa8UWYksW8ecZs1p04Wf kXsS2yu4Z/wFAY+o1oXpPCTly7xUsEoWjKNJQ/emzsjYIoxiN4xVCjVcnObKWh4uqKQtDg u/klidyns/gcir8jAqQUmEsTb6K1tv0N+VSBS4MCLRd7piAoj96Rah+Nv6e2MQNO5vZlKN Ao4yu0oplvHvTSRvnE0CcfgXBkDJtbEXC4AAg8KdUD3N3+KzthFMzTJ0NrdKFzo351zyVl ELzRQ7t/dAxy9wiyL5uAuBK4P662dwYdYWECVQrASJPBlT5ezNjdu2st1lm0UA== Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: sunpoet) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PB9nr6bNxzYb6 for ; Tue, 7 Feb 2023 17:57:36 +0000 (UTC) (envelope-from sunpoet@freebsd.org) Received: by mail-ed1-f41.google.com with SMTP id fi26so17106580edb.7 for ; Tue, 07 Feb 2023 09:57:36 -0800 (PST) X-Gm-Message-State: AO0yUKV+av+0MjhBbb0TNjsvu9/0diR2dim/thRW7B3JF9d1A+0iNMFA VNiENZNGw9sqvaeW8l3tBB+lAU7n7ZcnqxErCJa0ZQ== X-Google-Smtp-Source: AK7set9KV+0Wb/4wj7SPvMpk0Y6uaG7QCczfrJjzJ8BxaphynmiueFqhwF3QigBrBLiXZ9vvmOBTpIdjHJW4mdJDOdo= X-Received: by 2002:a50:f689:0:b0:4a3:fc00:9d69 with SMTP id d9-20020a50f689000000b004a3fc009d69mr901376edn.6.1675792655377; Tue, 07 Feb 2023 09:57:35 -0800 (PST) List-Id: Commit messages for all branches of the ports repository List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-ports-all@freebsd.org X-BeenThere: dev-commits-ports-all@freebsd.org MIME-Version: 1.0 References: <202302051941.315JfQOP071383@gitrepo.freebsd.org> In-Reply-To: <202302051941.315JfQOP071383@gitrepo.freebsd.org> From: Po-Chuan Hsieh Date: Wed, 8 Feb 2023 01:56:58 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode" To: Charlie Li Cc: ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Content-Type: multipart/alternative; boundary="000000000000f1007105f41fe1d2" X-ThisMailContainsUnwantedMimeParts: N --000000000000f1007105f41fe1d2 Content-Type: text/plain; charset="UTF-8" Hello, On Mon, Feb 6, 2023 at 3:41 AM Charlie Li wrote: > The branch main has been updated by vishwin: > > URL: > https://cgit.FreeBSD.org/ports/commit/?id=f5890bd3cbc66a805bfc04a1a45688b5a3e68c50 > > commit f5890bd3cbc66a805bfc04a1a45688b5a3e68c50 > Author: Charlie Li > AuthorDate: 2023-02-05 19:05:02 +0000 > Commit: Charlie Li > CommitDate: 2023-02-05 19:05:02 +0000 > > Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and > install bytecode" > Despite installer's default behaviour to compile and install bytecode, > we are not doing so going forward at stage/package time. [0] During > initial development and qualification of PEP-517 framework support, > compiling and installing bytecode at stage/package time was considered, > but was found problematic, fragile and ultimately unreliable, both > currently and historically (with USE_PYTHON=distutils), due to our > fixed plist requirement. While the living binary distribution format > (wheel) specification [1] says to compile bytecode, that is in the > pure Python package management context (pip, etc); nuance always > exists when interacting with "system" package management. > > Additionally, "bytecode is an implementation detail of the CPython > interpreter. No guarantees are made that bytecode will not be added, > removed, or changed between versions of Python," thus "should not > be considered to work across Python VMs or Python releases." [2] > This is important to ensuring correctness for those ports specifying > NO_ARCH. > Those issues you mentioned might be a problem in the future but clearly not in the short term. I'm trying to fix an immediate problem which greatly affects the development. It may not be good from your point of view but it really fixed the problem (see below). If you have a better solution, I'm fine with it. > > Instead of compiling and installing bytecode at stage/package time, > there is a WIP, review D34739, that compiles and installs bytecode > at install time instead, using triggers. > Thanks for pointing out a possible solution but it is still WIP. > The aforementioned build_fs_violations will be investigated. You could reproduce this problem as follows. 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the background) 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black here. Now it failed earlier with devel/py-click. You could expect more ports to be failed/skipped while building hundreds of ports in my case. % sudo poudriere bulk -j 12 -r -t devel/py-pyls-black [...] [00:00:10] Hit CTRL+t at any time to see build progress and stats [00:00:14] [01] [00:00:00] Building devel/py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0 [00:00:19] [01] [00:00:05] Finished devel/py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0: Success [00:00:19] [01] [00:00:00] Building devel/py-click@py39 | py39-click-8.1.3 [00:00:24] [01] [00:00:05] Finished devel/py-click@py39 | py39-click-8.1.3: Failed: stage_fs_violation [00:00:25] [01] [00:00:06] Skipping devel/py-black | py39-black-22.10.0_2: Dependent port devel/py-click | py39-click-8.1.3 failed [00:00:25] [01] [00:00:06] Skipping devel/py-pyls-black | py39-pyls-black-0.4.7_2: Dependent port devel/py-click | py39-click-8.1.3 failed [00:00:25] Stopping 4 builders [...] [00:01:36] Built ports: devel/py-importlib-metadata [00:01:36] Failed ports: devel/py-click:stage_fs_violation [00:01:36] Skipped ports: devel/py-black devel/py-pyls-black [...] from py39-click-8.1.3 log: [...] =>> Checking for staging violations... done =>> Error: Filesystem touched during stage (files must install to ${STAGEDIR}): extra: usr/local/lib/python3.9/site-packages/importlib_metadata/__pycache__ =>> Cleaning up wrkdir ===> Cleaning for py39-click-8.1.3 build of devel/py-click | py39-click-8.1.3 ended at Wed Feb 8 00:15:44 CST 2023 build time: 00:00:04 !!! build failure encountered !!! > This reverts commit de6965254c3a007efcf697c3d455b54d2aeb2383. > > With hat: python > Approved by: tcberner (mentor, portmgr) > Reference: https://wiki.freebsd.org/Python/PEP-517 [0] > > https://packaging.python.org/en/latest/specifications/binary-distribution-format/ > [1] > https://docs.python.org/3/library/dis.html [2] > --000000000000f1007105f41fe1d2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

On Mon, Feb 6, 2023 at 3:41 AM Charlie Li <vishwin@freebsd.org> wrot= e:
The branch main has been updated by vishwin:

URL: https://cgi= t.FreeBSD.org/ports/commit/?id=3Df5890bd3cbc66a805bfc04a1a45688b5a3e68c50

commit f5890bd3cbc66a805bfc04a1a45688b5a3e68c50
Author:=C2=A0 =C2=A0 =C2=A0Charlie Li <vishwin@FreeBSD.org>
AuthorDate: 2023-02-05 19:05:02 +0000
Commit:=C2=A0 =C2=A0 =C2=A0Charlie Li <vishwin@FreeBSD.org>
CommitDate: 2023-02-05 19:05:02 +0000

=C2=A0 =C2=A0 Revert "Mk/Uses/python.mk: Fix USE_PYTHON=3Dpep517: always co= mpile and install bytecode"=C2=A0

=C2=A0 =C2=A0 Despite installer's default behaviour to compile and inst= all bytecode,
=C2=A0 =C2=A0 we are not doing so going forward at stage/package time. [0] = During
=C2=A0 =C2=A0 initial development and qualification of PEP-517 framework su= pport,
=C2=A0 =C2=A0 compiling and installing bytecode at stage/package time was c= onsidered,
=C2=A0 =C2=A0 but was found problematic, fragile and ultimately unreliable,= both
=C2=A0 =C2=A0 currently and historically (with USE_PYTHON=3Ddistutils), due= to our
=C2=A0 =C2=A0 fixed plist requirement. While the living binary distribution= format
=C2=A0 =C2=A0 (wheel) specification [1] says to compile bytecode, that is i= n the
=C2=A0 =C2=A0 pure Python package management context (pip, etc); nuance alw= ays
=C2=A0 =C2=A0 exists when interacting with "system" package manag= ement.

=C2=A0 =C2=A0 Additionally, "bytecode is an implementation detail of t= he CPython
=C2=A0 =C2=A0 interpreter. No guarantees are made that bytecode will not be= added,
=C2=A0 =C2=A0 removed, or changed between versions of Python," thus &q= uot;should not
=C2=A0 =C2=A0 be considered to work across Python VMs or Python releases.&q= uot; [2]
=C2=A0 =C2=A0 This is important to ensuring correctness for those ports spe= cifying
=C2=A0 =C2=A0 NO_ARCH.

Those issue= s you mentioned might be a problem in the=C2=A0future but clearly not in th= e=C2=A0short term.
I'm trying to fix an=C2=A0immediate proble= m which greatly affects the development.
It m= ay not be good from=C2=A0your point of view but it=C2=A0really fixed the pr= oblem (see below).
If you have a better solution, I&#= 39;m fine with it.
=C2=A0

=C2=A0 =C2=A0 Instead of compiling and installing bytecode at stage/package= time,
=C2=A0 =C2=A0 there is a WIP, review D34739, that compiles and installs byt= ecode
=C2=A0 =C2=A0 at install time instead, using triggers.

Thanks for pointing out a possible solution but it is still= WIP.


=C2=A0 =C2=A0 The aforementioned build_fs_violations will be investigated.<= /blockquote>

<= /div>
You could reproduce this problem as follows.
1. Revert= =C2=A0d6583d67b69dbf690e6f102e1aec799cb7fd564a (change devel/py-importlib-m= etadata back to USE_PYTHON=3Dpep517 which is the background)
2. B= uild a port with USE_PYTHON=3Dpep517. I choose devel/py-pyls-black here.
Now it failed earlier with devel/py-click.
You could expe= ct more ports to be failed/skipped while building hundreds of ports in my c= ase.

%=C2=A0sudo poudriere bulk -j 12 -r -t devel/= py-pyls-black
[...]
[00:00:10] Hit CTRL+t at any time t= o see build progress and stats
[00:00:14] [01] [00:00:00] Building devel= /py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0
[00:00:19] [= 01] [00:00:05] Finished devel/py-importlib-metadata@py39 | py39-importlib-m= etadata-6.0.0: Success
[00:00:19] [01] [00:00:00] Building devel/py-clic= k@py39 | py39-click-8.1.3
[00:00:24] [01] [00:00:05] Finished devel/py-c= lick@py39 | py39-click-8.1.3: Failed: stage_fs_violation
[00:00:25] [01]= [00:00:06] Skipping devel/py-black | py39-black-22.10.0_2: Dependent port = devel/py-click | py39-click-8.1.3 failed
[00:00:25] [01] [00:00:06] Skip= ping devel/py-pyls-black | py39-pyls-black-0.4.7_2: Dependent port devel/py= -click | py39-click-8.1.3 failed
[00:00:25] Stopping 4 builders
[...]
[00:01:36] Built ports: devel/py-importlib-metadata[00:01:36] Failed ports: devel/py-click:stage_fs_violation
[00:01:36] = Skipped ports: devel/py-black devel/py-pyls-black
[...]
=

from py39-click-8.1.3 log:
[...]
= =3D>> Checking for staging violations... done
=3D>> Error: F= ilesystem touched during stage (files must install to ${STAGEDIR}):
extr= a: usr/local/lib/python3.9/site-packages/importlib_metadata/__pycache__
= =3D>> Cleaning up wrkdir
=3D=3D=3D> =C2=A0Cleaning for py39-cli= ck-8.1.3
build of devel/py-click | py39-click-8.1.3 ended at Wed Feb =C2= =A08 00:15:44 CST 2023
build time: 00:00:04
!!! build failure encount= ered !!!


=C2=A0 =C2=A0 This reverts commit de6965254c3a007efcf697c3d455b54d2aeb2383.=

=C2=A0 =C2=A0 With hat:=C2=A0 =C2=A0 =C2=A0 =C2=A0python
=C2=A0 =C2=A0 Approved by:=C2=A0 =C2=A0 tcberner (mentor, portmgr)
=C2=A0 =C2=A0 Reference:=C2=A0 =C2=A0 =C2=A0 https://wiki.fre= ebsd.org/Python/PEP-517 [0]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 https://packaging.python= .org/en/latest/specifications/binary-distribution-format/ [1]
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 https://docs.python.org/3/library/dis.html [2]
--000000000000f1007105f41fe1d2--