Date: Wed, 8 Feb 2023 01:56:58 +0800 From: Po-Chuan Hsieh <sunpoet@freebsd.org> To: Charlie Li <vishwin@freebsd.org> Cc: ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Subject: Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode" Message-ID: <CAMHz58QuYSdSi0J93aLoBbjfR3ncizOjFcYyXizUZ%2BP0hji_pg@mail.gmail.com> In-Reply-To: <202302051941.315JfQOP071383@gitrepo.freebsd.org> References: <202302051941.315JfQOP071383@gitrepo.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000f1007105f41fe1d2 Content-Type: text/plain; charset="UTF-8" Hello, On Mon, Feb 6, 2023 at 3:41 AM Charlie Li <vishwin@freebsd.org> wrote: > The branch main has been updated by vishwin: > > URL: > https://cgit.FreeBSD.org/ports/commit/?id=f5890bd3cbc66a805bfc04a1a45688b5a3e68c50 > > commit f5890bd3cbc66a805bfc04a1a45688b5a3e68c50 > Author: Charlie Li <vishwin@FreeBSD.org> > AuthorDate: 2023-02-05 19:05:02 +0000 > Commit: Charlie Li <vishwin@FreeBSD.org> > CommitDate: 2023-02-05 19:05:02 +0000 > > Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and > install bytecode" > Despite installer's default behaviour to compile and install bytecode, > we are not doing so going forward at stage/package time. [0] During > initial development and qualification of PEP-517 framework support, > compiling and installing bytecode at stage/package time was considered, > but was found problematic, fragile and ultimately unreliable, both > currently and historically (with USE_PYTHON=distutils), due to our > fixed plist requirement. While the living binary distribution format > (wheel) specification [1] says to compile bytecode, that is in the > pure Python package management context (pip, etc); nuance always > exists when interacting with "system" package management. > > Additionally, "bytecode is an implementation detail of the CPython > interpreter. No guarantees are made that bytecode will not be added, > removed, or changed between versions of Python," thus "should not > be considered to work across Python VMs or Python releases." [2] > This is important to ensuring correctness for those ports specifying > NO_ARCH. > Those issues you mentioned might be a problem in the future but clearly not in the short term. I'm trying to fix an immediate problem which greatly affects the development. It may not be good from your point of view but it really fixed the problem (see below). If you have a better solution, I'm fine with it. > > Instead of compiling and installing bytecode at stage/package time, > there is a WIP, review D34739, that compiles and installs bytecode > at install time instead, using triggers. > Thanks for pointing out a possible solution but it is still WIP. > The aforementioned build_fs_violations will be investigated. You could reproduce this problem as follows. 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the background) 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black here. Now it failed earlier with devel/py-click. You could expect more ports to be failed/skipped while building hundreds of ports in my case. % sudo poudriere bulk -j 12 -r -t devel/py-pyls-black [...] [00:00:10] Hit CTRL+t at any time to see build progress and stats [00:00:14] [01] [00:00:00] Building devel/py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0 [00:00:19] [01] [00:00:05] Finished devel/py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0: Success [00:00:19] [01] [00:00:00] Building devel/py-click@py39 | py39-click-8.1.3 [00:00:24] [01] [00:00:05] Finished devel/py-click@py39 | py39-click-8.1.3: Failed: stage_fs_violation [00:00:25] [01] [00:00:06] Skipping devel/py-black | py39-black-22.10.0_2: Dependent port devel/py-click | py39-click-8.1.3 failed [00:00:25] [01] [00:00:06] Skipping devel/py-pyls-black | py39-pyls-black-0.4.7_2: Dependent port devel/py-click | py39-click-8.1.3 failed [00:00:25] Stopping 4 builders [...] [00:01:36] Built ports: devel/py-importlib-metadata [00:01:36] Failed ports: devel/py-click:stage_fs_violation [00:01:36] Skipped ports: devel/py-black devel/py-pyls-black [...] from py39-click-8.1.3 log: [...] =>> Checking for staging violations... done =>> Error: Filesystem touched during stage (files must install to ${STAGEDIR}): extra: usr/local/lib/python3.9/site-packages/importlib_metadata/__pycache__ =>> Cleaning up wrkdir ===> Cleaning for py39-click-8.1.3 build of devel/py-click | py39-click-8.1.3 ended at Wed Feb 8 00:15:44 CST 2023 build time: 00:00:04 !!! build failure encountered !!! > This reverts commit de6965254c3a007efcf697c3d455b54d2aeb2383. > > With hat: python > Approved by: tcberner (mentor, portmgr) > Reference: https://wiki.freebsd.org/Python/PEP-517 [0] > > https://packaging.python.org/en/latest/specifications/binary-distribution-format/ > [1] > https://docs.python.org/3/library/dis.html [2] > --000000000000f1007105f41fe1d2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr">Hello,</div><div dir=3D"ltr"><br></div><d= iv dir=3D"ltr">On Mon, Feb 6, 2023 at 3:41 AM Charlie Li <<a href=3D"mai= lto:vishwin@freebsd.org" target=3D"_blank">vishwin@freebsd.org</a>> wrot= e:<br></div><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" st= yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd= ing-left:1ex">The branch main has been updated by vishwin:<br> <br> URL: <a href=3D"https://cgit.FreeBSD.org/ports/commit/?id=3Df5890bd3cbc66a8= 05bfc04a1a45688b5a3e68c50" rel=3D"noreferrer" target=3D"_blank">https://cgi= t.FreeBSD.org/ports/commit/?id=3Df5890bd3cbc66a805bfc04a1a45688b5a3e68c50</= a><br> <br> commit f5890bd3cbc66a805bfc04a1a45688b5a3e68c50<br> Author:=C2=A0 =C2=A0 =C2=A0Charlie Li <vishwin@FreeBSD.org><br> AuthorDate: 2023-02-05 19:05:02 +0000<br> Commit:=C2=A0 =C2=A0 =C2=A0Charlie Li <vishwin@FreeBSD.org><br> CommitDate: 2023-02-05 19:05:02 +0000<br> <br> =C2=A0 =C2=A0 Revert "Mk/Uses/<a href=3D"http://python.mk" rel=3D"nore= ferrer" target=3D"_blank">python.mk</a>: Fix USE_PYTHON=3Dpep517: always co= mpile and install bytecode"=C2=A0</blockquote><blockquote class=3D"gma= il_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,2= 04,204);padding-left:1ex"> <br> =C2=A0 =C2=A0 Despite installer's default behaviour to compile and inst= all bytecode,<br> =C2=A0 =C2=A0 we are not doing so going forward at stage/package time. [0] = During<br> =C2=A0 =C2=A0 initial development and qualification of PEP-517 framework su= pport,<br> =C2=A0 =C2=A0 compiling and installing bytecode at stage/package time was c= onsidered,<br> =C2=A0 =C2=A0 but was found problematic, fragile and ultimately unreliable,= both<br> =C2=A0 =C2=A0 currently and historically (with USE_PYTHON=3Ddistutils), due= to our<br> =C2=A0 =C2=A0 fixed plist requirement. While the living binary distribution= format<br> =C2=A0 =C2=A0 (wheel) specification [1] says to compile bytecode, that is i= n the<br> =C2=A0 =C2=A0 pure Python package management context (pip, etc); nuance alw= ays<br> =C2=A0 =C2=A0 exists when interacting with "system" package manag= ement.<br> <br> =C2=A0 =C2=A0 Additionally, "bytecode is an implementation detail of t= he CPython<br> =C2=A0 =C2=A0 interpreter. No guarantees are made that bytecode will not be= added,<br> =C2=A0 =C2=A0 removed, or changed between versions of Python," thus &q= uot;should not<br> =C2=A0 =C2=A0 be considered to work across Python VMs or Python releases.&q= uot; [2]<br> =C2=A0 =C2=A0 This is important to ensuring correctness for those ports spe= cifying<br> =C2=A0 =C2=A0 NO_ARCH.<br></blockquote><div><br></div><div><div>Those issue= s you mentioned might be a problem in the=C2=A0future but clearly not in th= e=C2=A0short term.</div><div>I'm trying to fix an=C2=A0immediate proble= m which greatly affects the development.<br></div></div><div><div><div>It m= ay not be good from=C2=A0your point of view but it=C2=A0really fixed the pr= oblem (see below).</div></div></div><div>If you have a better solution, I&#= 39;m fine with it.</div><div>=C2=A0<br></div><blockquote class=3D"gmail_quo= te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204= );padding-left:1ex"> <br> =C2=A0 =C2=A0 Instead of compiling and installing bytecode at stage/package= time,<br> =C2=A0 =C2=A0 there is a WIP, review D34739, that compiles and installs byt= ecode<br> =C2=A0 =C2=A0 at install time instead, using triggers.<br></blockquote><div= ><br></div><div>Thanks for pointing out a possible solution but it is still= WIP.</div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin= :0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"= > <br> =C2=A0 =C2=A0 The aforementioned build_fs_violations will be investigated.<= /blockquote><div><div><br class=3D"gmail-Apple-interchange-newline"></div><= /div><div>You could reproduce this problem as follows.</div><div>1. Revert= =C2=A0d6583d67b69dbf690e6f102e1aec799cb7fd564a (change devel/py-importlib-m= etadata back to USE_PYTHON=3Dpep517 which is the background)</div><div>2. B= uild a port with USE_PYTHON=3Dpep517. I choose devel/py-pyls-black here.</d= iv><div>Now it failed earlier with devel/py-click.</div><div>You could expe= ct more ports to be failed/skipped while building hundreds of ports in my c= ase.</div><div><br></div><div>%=C2=A0sudo poudriere bulk -j 12 -r -t devel/= py-pyls-black</div><div>[...]</div><div>[00:00:10] Hit CTRL+t at any time t= o see build progress and stats<br>[00:00:14] [01] [00:00:00] Building devel= /py-importlib-metadata@py39 | py39-importlib-metadata-6.0.0<br>[00:00:19] [= 01] [00:00:05] Finished devel/py-importlib-metadata@py39 | py39-importlib-m= etadata-6.0.0: Success<br>[00:00:19] [01] [00:00:00] Building devel/py-clic= k@py39 | py39-click-8.1.3<br>[00:00:24] [01] [00:00:05] Finished devel/py-c= lick@py39 | py39-click-8.1.3: Failed: stage_fs_violation<br>[00:00:25] [01]= [00:00:06] Skipping devel/py-black | py39-black-22.10.0_2: Dependent port = devel/py-click | py39-click-8.1.3 failed<br>[00:00:25] [01] [00:00:06] Skip= ping devel/py-pyls-black | py39-pyls-black-0.4.7_2: Dependent port devel/py= -click | py39-click-8.1.3 failed<br>[00:00:25] Stopping 4 builders<br></div= ><div>[...]</div><div>[00:01:36] Built ports: devel/py-importlib-metadata<b= r>[00:01:36] Failed ports: devel/py-click:stage_fs_violation<br>[00:01:36] = Skipped ports: devel/py-black devel/py-pyls-black<br></div><div>[...]</div>= <div><br></div><div>from py39-click-8.1.3 log:</div><div>[...]</div><div>= =3D>> Checking for staging violations... done<br>=3D>> Error: F= ilesystem touched during stage (files must install to ${STAGEDIR}):<br>extr= a: usr/local/lib/python3.9/site-packages/importlib_metadata/__pycache__<br>= =3D>> Cleaning up wrkdir<br>=3D=3D=3D> =C2=A0Cleaning for py39-cli= ck-8.1.3<br>build of devel/py-click | py39-click-8.1.3 ended at Wed Feb =C2= =A08 00:15:44 CST 2023<br>build time: 00:00:04<br>!!! build failure encount= ered !!!<br></div><div><br></div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le= ft:1ex"> <br> =C2=A0 =C2=A0 This reverts commit de6965254c3a007efcf697c3d455b54d2aeb2383.= <br> <br> =C2=A0 =C2=A0 With hat:=C2=A0 =C2=A0 =C2=A0 =C2=A0python<br> =C2=A0 =C2=A0 Approved by:=C2=A0 =C2=A0 tcberner (mentor, portmgr)<br> =C2=A0 =C2=A0 Reference:=C2=A0 =C2=A0 =C2=A0 <a href=3D"https://wiki.freebs= d.org/Python/PEP-517" rel=3D"noreferrer" target=3D"_blank">https://wiki.fre= ebsd.org/Python/PEP-517</a> [0]<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a hr= ef=3D"https://packaging.python.org/en/latest/specifications/binary-distribu= tion-format/" rel=3D"noreferrer" target=3D"_blank">https://packaging.python= .org/en/latest/specifications/binary-distribution-format/</a> [1]<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a hr= ef=3D"https://docs.python.org/3/library/dis.html" rel=3D"noreferrer" target= =3D"_blank">https://docs.python.org/3/library/dis.html</a>; [2]<br> </blockquote></div></div> --000000000000f1007105f41fe1d2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMHz58QuYSdSi0J93aLoBbjfR3ncizOjFcYyXizUZ%2BP0hji_pg>