Date: Wed, 8 Feb 2023 03:41:16 +0800 From: Po-Chuan Hsieh <sunpoet@freebsd.org> To: Charlie Li <vishwin@freebsd.org> Cc: ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Subject: Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode" Message-ID: <CAMHz58St0=QNVh9BM7pxqsibSJzPFuC-FYAvHhdL=qukBWWhXQ@mail.gmail.com> In-Reply-To: <729ca691-4f25-a094-a6b4-99490674ecd0@freebsd.org> References: <202302051941.315JfQOP071383@gitrepo.freebsd.org> <CAMHz58QuYSdSi0J93aLoBbjfR3ncizOjFcYyXizUZ%2BP0hji_pg@mail.gmail.com> <729ca691-4f25-a094-a6b4-99490674ecd0@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000007041a05f42157ca Content-Type: text/plain; charset="UTF-8" On Wed, Feb 8, 2023 at 2:36 AM Charlie Li <vishwin@freebsd.org> wrote: > Po-Chuan Hsieh wrote: > > Those issues you mentioned might be a problem in the future but clearly > > not in the short term. > > I'm trying to fix an immediate problem which greatly affects the > > development. > They are problems now, because they impede any proper fixes. We have > been doing quite a few things wrong with Python packaging, and simply > fixing "immediate problems" doesn't work. > > It may not be good from your point of view but it really fixed the > > problem (see below). > > If you have a better solution, I'm fine with it. > > > > > > Instead of compiling and installing bytecode at stage/package > time, > > there is a WIP, review D34739, that compiles and installs > bytecode > > at install time instead, using triggers. > > > > > > Thanks for pointing out a possible solution but it is still WIP. > > > It will be *the* solution, because bytecode is not meant to be packaged, > but only compiled at install or run-time at user's discretion. The only > reason why bytecode was packaged in operating system-level package > distributions is how most such systems lack a trigger function that we > now have. > When Python packaging tooling is the final arbiter, bytecode is compiled > only after the installation process completes if the user does not > disable it. In our case, pkg(8) is the final arbiter, so such tasking > falls to pkg(8)'s triggers and not during stage/packaging. > > > > The aforementioned build_fs_violations will be investigated. > > > > > > You could reproduce this problem as follows. > > 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change > > devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the > > background) > > 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black > here. > > Now it failed earlier with devel/py-click. > > You could expect more ports to be failed/skipped while building hundreds > > of ports in my case. > > > Can't reproduce with a proper PEP-517 bootstrap tooling, which most > importantly does not have setuptools in the environment at all. If > anything, your build failure only demonstrates just how "problematic, > fragile and unreliable" compiling and installing bytecode at > stage/package time is. > The problem exists. I'm not insisting on putting bytecode in the package. But we need a fix or hack before the solution (trigger) lands. Let me explain it in more detail. While building devel/py-installer, PEP517_BUILD_CMD executes flit_core.wheel and the corresponding bytecodes are generated in the __pycache__ directories. The fs outside STAGEDIR is being changed. It makes build_fs_violation failure in poudriere. There is nothing about setuptools. I wonder if you really tested it. This is the up-to-date example (a fresh checkout). % git st devel/py-flit-core devel/py-installer On branch main Your branch is up to date with 'freebsd/main'. nothing to commit, working tree clean % sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core devel/py-installer [...] [00:02:11] [01] [00:00:00] Building lang/python39 | python39-3.9.16 [00:03:13] [01] [00:01:02] Finished lang/python39 | python39-3.9.16: Success [00:03:14] [01] [00:00:00] Building devel/py-flit-core@py39 | py39-flit-core-3.8.0_1 [00:03:17] [01] [00:00:03] Finished devel/py-flit-core@py39 | py39-flit-core-3.8.0_1: Success [00:03:17] [01] [00:00:00] Building devel/py-installer@py39 | py39-installer-0.6.0_1 [00:03:21] [01] [00:00:04] Finished devel/py-installer@py39 | py39-installer-0.6.0_1: Failed: build_fs_violation [...] [00:00:44] Built ports: devel/py-flit-core [00:00:44] Failed ports: devel/py-installer:build_fs_violation [12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2 Built: 1 Failed: 1 Skipped: 0 Ignored: 0 Tobuild: 0 Time: 00:00:44 [...] % tail -10 py39-installer-0.6.0_1.log =>> Checking for filesystem violations... done =>> Error: Filesystem touched during build: extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/tomli/__pycache__ extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/__pycache__ extra: usr/local/lib/python3.9/site-packages/flit_core/__pycache__ =>> Cleaning up wrkdir ===> Cleaning for py39-installer-0.6.0_1 build of devel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb 8 03:11:24 CST 2023 build time: 00:00:02 !!! build failure encountered !!! % grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log % grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log | wc -l 0 --00000000000007041a05f42157ca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr">On Wed, Feb 8, 2023 at 2:36 AM Charlie Li= <<a href=3D"mailto:vishwin@freebsd.org" target=3D"_blank">vishwin@freeb= sd.org</a>> wrote:<br></div><div class=3D"gmail_quote"><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex">Po-Chuan Hsieh wrote:<br> > Those issues you mentioned might be a problem in the=C2=A0future but c= learly <br> > not in the=C2=A0short term.<br> > I'm trying to fix an=C2=A0immediate problem which greatly affects = the <br> > development.<br> They are problems now, because they impede any proper fixes. We have <br> been doing quite a few things wrong with Python packaging, and simply <br> fixing "immediate problems" doesn't work.<br> > It may not be good from=C2=A0your point of view but it=C2=A0really fix= ed the <br> > problem (see below).<br> > If you have a better solution, I'm fine with it.<br> > <br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Instead of compiling and installing = bytecode at stage/package time,<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 there is a WIP, review D34739, that = compiles and installs bytecode<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at install time instead, using trigg= ers.<br> > <br> > <br> > Thanks for pointing out a possible solution but it is still WIP.<br> > <br> It will be *the* solution, because bytecode is not meant to be packaged, <b= r> but only compiled at install or run-time at user's discretion. The only= <br> reason why bytecode was packaged in operating system-level package <br> distributions is how most such systems lack a trigger function that we <br> now have.=C2=A0</blockquote><blockquote class=3D"gmail_quote" style=3D"marg= in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e= x"> <br> When Python packaging tooling is the final arbiter, bytecode is compiled <b= r> only after the installation process completes if the user does not <br> disable it. In our case, pkg(8) is the final arbiter, so such tasking <br> falls to pkg(8)'s triggers and not during stage/packaging.<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 The aforementioned build_fs_violatio= ns will be investigated.<br> > <br> > <br> > You could reproduce this problem as follows.<br> > 1. Revert=C2=A0d6583d67b69dbf690e6f102e1aec799cb7fd564a (change <br> > devel/py-importlib-metadata back to USE_PYTHON=3Dpep517 which is the <= br> > background)<br> > 2. Build a port with USE_PYTHON=3Dpep517. I choose devel/py-pyls-black= here.<br> > Now it failed earlier with devel/py-click.<br> > You could expect more ports to be failed/skipped while building hundre= ds <br> > of ports in my case.<br> > <br> Can't reproduce with a proper PEP-517 bootstrap tooling, which most <br= > importantly does not have setuptools in the environment at all. If <br> anything, your build failure only demonstrates just how "problematic, = <br> fragile and unreliable" compiling and installing bytecode at <br> stage/package time is.<br></blockquote><div><br></div><div>The problem exis= ts.</div><div><div>I'm not insisting=C2=A0on putting=C2=A0bytecode in t= he package.</div><div>But we need a fix or hack before the solution (trigge= r) lands.</div><div><br></div><div><div>Let me explain it in more detail.</= div></div><div>While building devel/py-installer, PEP517_BUILD_CMD executes= flit_core.wheel and the corresponding bytecodes are generated in the __pyc= ache__ directories.<br></div><div>The fs outside STAGEDIR is being changed.= </div><div>It makes build_fs_violation failure in poudriere.</div></div><di= v>There is nothing about setuptools.</div><div><br></div><div><div>I wonder= if you really tested it.</div><div>This is the=C2=A0up-to-date example (a = fresh checkout).</div></div><div><br></div><div>% git st devel/py-flit-core= devel/py-installer<br></div>On branch main<br>Your branch is up to date wi= th 'freebsd/main'.<br><br>nothing to commit, working tree clean<br>= % sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core devel/py-in= staller<br>[...]<br>[00:02:11] [01] [00:00:00] Building lang/python39 | pyt= hon39-3.9.16<br>[00:03:13] [01] [00:01:02] Finished lang/python39 | python3= 9-3.9.16: Success<br>[00:03:14] [01] [00:00:00] Building devel/py-flit-core= @py39 | py39-flit-core-3.8.0_1<br>[00:03:17] [01] [00:00:03] Finished devel= /py-flit-core@py39 | py39-flit-core-3.8.0_1: Success<br>[00:03:17] [01] [00= :00:00] Building devel/py-installer@py39 | py39-installer-0.6.0_1<br>[00:03= :21] [01] [00:00:04] Finished devel/py-installer@py39 | py39-installer-0.6.= 0_1: Failed: build_fs_violation<br>[...]<br>[00:00:44] Built ports: devel/p= y-flit-core<br>[00:00:44] Failed ports: devel/py-installer:build_fs_violati= on<br>[12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2 =C2=A0Bui= lt: 1 =C2=A0Failed: 1 =C2=A0Skipped: 0 =C2=A0Ignored: 0 =C2=A0Tobuild: 0 = =C2=A0 Time: 00:00:44<br>[...]<br><br>% tail -10 py39-installer-0.6.0_1.log= <br>=3D>> Checking for filesystem violations... done<br>=3D>> E= rror: Filesystem touched during build:<br>extra: usr/local/lib/python3.9/si= te-packages/flit_core/vendor/tomli/__pycache__<br>extra: usr/local/lib/pyth= on3.9/site-packages/flit_core/vendor/__pycache__<br>extra: usr/local/lib/py= thon3.9/site-packages/flit_core/__pycache__<br>=3D>> Cleaning up wrkd= ir<br>=3D=3D=3D> =C2=A0Cleaning for py39-installer-0.6.0_1<br>build of d= evel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb =C2=A08 03:11:2= 4 CST 2023<br>build time: 00:00:02<br>!!! build failure encountered !!!<br>= % grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log<br>= % grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log | w= c -l<br><div>=C2=A0 =C2=A0 =C2=A0 =C2=A00</div></div></div> --00000000000007041a05f42157ca--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMHz58St0=QNVh9BM7pxqsibSJzPFuC-FYAvHhdL=qukBWWhXQ>