Date: Wed, 8 Feb 2023 03:41:16 +0800 From: Po-Chuan Hsieh <sunpoet@freebsd.org> To: Charlie Li <vishwin@freebsd.org> Cc: ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Subject: Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode" Message-ID: <CAMHz58St0=QNVh9BM7pxqsibSJzPFuC-FYAvHhdL=qukBWWhXQ@mail.gmail.com> In-Reply-To: <729ca691-4f25-a094-a6b4-99490674ecd0@freebsd.org> References: <202302051941.315JfQOP071383@gitrepo.freebsd.org> <CAMHz58QuYSdSi0J93aLoBbjfR3ncizOjFcYyXizUZ%2BP0hji_pg@mail.gmail.com> <729ca691-4f25-a094-a6b4-99490674ecd0@freebsd.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
On Wed, Feb 8, 2023 at 2:36 AM Charlie Li <vishwin@freebsd.org> wrote:
> Po-Chuan Hsieh wrote:
> > Those issues you mentioned might be a problem in the future but clearly
> > not in the short term.
> > I'm trying to fix an immediate problem which greatly affects the
> > development.
> They are problems now, because they impede any proper fixes. We have
> been doing quite a few things wrong with Python packaging, and simply
> fixing "immediate problems" doesn't work.
> > It may not be good from your point of view but it really fixed the
> > problem (see below).
> > If you have a better solution, I'm fine with it.
> >
> >
> > Instead of compiling and installing bytecode at stage/package
> time,
> > there is a WIP, review D34739, that compiles and installs
> bytecode
> > at install time instead, using triggers.
> >
> >
> > Thanks for pointing out a possible solution but it is still WIP.
> >
> It will be *the* solution, because bytecode is not meant to be packaged,
> but only compiled at install or run-time at user's discretion. The only
> reason why bytecode was packaged in operating system-level package
> distributions is how most such systems lack a trigger function that we
> now have.
> When Python packaging tooling is the final arbiter, bytecode is compiled
> only after the installation process completes if the user does not
> disable it. In our case, pkg(8) is the final arbiter, so such tasking
> falls to pkg(8)'s triggers and not during stage/packaging.
> >
> > The aforementioned build_fs_violations will be investigated.
> >
> >
> > You could reproduce this problem as follows.
> > 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change
> > devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the
> > background)
> > 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black
> here.
> > Now it failed earlier with devel/py-click.
> > You could expect more ports to be failed/skipped while building hundreds
> > of ports in my case.
> >
> Can't reproduce with a proper PEP-517 bootstrap tooling, which most
> importantly does not have setuptools in the environment at all. If
> anything, your build failure only demonstrates just how "problematic,
> fragile and unreliable" compiling and installing bytecode at
> stage/package time is.
>
The problem exists.
I'm not insisting on putting bytecode in the package.
But we need a fix or hack before the solution (trigger) lands.
Let me explain it in more detail.
While building devel/py-installer, PEP517_BUILD_CMD executes
flit_core.wheel and the corresponding bytecodes are generated in the
__pycache__ directories.
The fs outside STAGEDIR is being changed.
It makes build_fs_violation failure in poudriere.
There is nothing about setuptools.
I wonder if you really tested it.
This is the up-to-date example (a fresh checkout).
% git st devel/py-flit-core devel/py-installer
On branch main
Your branch is up to date with 'freebsd/main'.
nothing to commit, working tree clean
% sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core
devel/py-installer
[...]
[00:02:11] [01] [00:00:00] Building lang/python39 | python39-3.9.16
[00:03:13] [01] [00:01:02] Finished lang/python39 | python39-3.9.16: Success
[00:03:14] [01] [00:00:00] Building devel/py-flit-core@py39 |
py39-flit-core-3.8.0_1
[00:03:17] [01] [00:00:03] Finished devel/py-flit-core@py39 |
py39-flit-core-3.8.0_1: Success
[00:03:17] [01] [00:00:00] Building devel/py-installer@py39 |
py39-installer-0.6.0_1
[00:03:21] [01] [00:00:04] Finished devel/py-installer@py39 |
py39-installer-0.6.0_1: Failed: build_fs_violation
[...]
[00:00:44] Built ports: devel/py-flit-core
[00:00:44] Failed ports: devel/py-installer:build_fs_violation
[12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2 Built: 1
Failed: 1 Skipped: 0 Ignored: 0 Tobuild: 0 Time: 00:00:44
[...]
% tail -10 py39-installer-0.6.0_1.log
=>> Checking for filesystem violations... done
=>> Error: Filesystem touched during build:
extra:
usr/local/lib/python3.9/site-packages/flit_core/vendor/tomli/__pycache__
extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/__pycache__
extra: usr/local/lib/python3.9/site-packages/flit_core/__pycache__
=>> Cleaning up wrkdir
===> Cleaning for py39-installer-0.6.0_1
build of devel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb 8
03:11:24 CST 2023
build time: 00:00:02
!!! build failure encountered !!!
% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log
% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log |
wc -l
0
[-- Attachment #2 --]
<div dir="ltr"><div dir="ltr">On Wed, Feb 8, 2023 at 2:36 AM Charlie Li <<a href="mailto:vishwin@freebsd.org" target="_blank">vishwin@freebsd.org</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Po-Chuan Hsieh wrote:<br>
> Those issues you mentioned might be a problem in the future but clearly <br>
> not in the short term.<br>
> I'm trying to fix an immediate problem which greatly affects the <br>
> development.<br>
They are problems now, because they impede any proper fixes. We have <br>
been doing quite a few things wrong with Python packaging, and simply <br>
fixing "immediate problems" doesn't work.<br>
> It may not be good from your point of view but it really fixed the <br>
> problem (see below).<br>
> If you have a better solution, I'm fine with it.<br>
> <br>
> <br>
> Instead of compiling and installing bytecode at stage/package time,<br>
> there is a WIP, review D34739, that compiles and installs bytecode<br>
> at install time instead, using triggers.<br>
> <br>
> <br>
> Thanks for pointing out a possible solution but it is still WIP.<br>
> <br>
It will be *the* solution, because bytecode is not meant to be packaged, <br>
but only compiled at install or run-time at user's discretion. The only <br>
reason why bytecode was packaged in operating system-level package <br>
distributions is how most such systems lack a trigger function that we <br>
now have. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
When Python packaging tooling is the final arbiter, bytecode is compiled <br>
only after the installation process completes if the user does not <br>
disable it. In our case, pkg(8) is the final arbiter, so such tasking <br>
falls to pkg(8)'s triggers and not during stage/packaging.<br>
> <br>
> The aforementioned build_fs_violations will be investigated.<br>
> <br>
> <br>
> You could reproduce this problem as follows.<br>
> 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change <br>
> devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the <br>
> background)<br>
> 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black here.<br>
> Now it failed earlier with devel/py-click.<br>
> You could expect more ports to be failed/skipped while building hundreds <br>
> of ports in my case.<br>
> <br>
Can't reproduce with a proper PEP-517 bootstrap tooling, which most <br>
importantly does not have setuptools in the environment at all. If <br>
anything, your build failure only demonstrates just how "problematic, <br>
fragile and unreliable" compiling and installing bytecode at <br>
stage/package time is.<br></blockquote><div><br></div><div>The problem exists.</div><div><div>I'm not insisting on putting bytecode in the package.</div><div>But we need a fix or hack before the solution (trigger) lands.</div><div><br></div><div><div>Let me explain it in more detail.</div></div><div>While building devel/py-installer, PEP517_BUILD_CMD executes flit_core.wheel and the corresponding bytecodes are generated in the __pycache__ directories.<br></div><div>The fs outside STAGEDIR is being changed.</div><div>It makes build_fs_violation failure in poudriere.</div></div><div>There is nothing about setuptools.</div><div><br></div><div><div>I wonder if you really tested it.</div><div>This is the up-to-date example (a fresh checkout).</div></div><div><br></div><div>% git st devel/py-flit-core devel/py-installer<br></div>On branch main<br>Your branch is up to date with 'freebsd/main'.<br><br>nothing to commit, working tree clean<br>% sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core devel/py-installer<br>[...]<br>[00:02:11] [01] [00:00:00] Building lang/python39 | python39-3.9.16<br>[00:03:13] [01] [00:01:02] Finished lang/python39 | python39-3.9.16: Success<br>[00:03:14] [01] [00:00:00] Building devel/py-flit-core@py39 | py39-flit-core-3.8.0_1<br>[00:03:17] [01] [00:00:03] Finished devel/py-flit-core@py39 | py39-flit-core-3.8.0_1: Success<br>[00:03:17] [01] [00:00:00] Building devel/py-installer@py39 | py39-installer-0.6.0_1<br>[00:03:21] [01] [00:00:04] Finished devel/py-installer@py39 | py39-installer-0.6.0_1: Failed: build_fs_violation<br>[...]<br>[00:00:44] Built ports: devel/py-flit-core<br>[00:00:44] Failed ports: devel/py-installer:build_fs_violation<br>[12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2 Built: 1 Failed: 1 Skipped: 0 Ignored: 0 Tobuild: 0 Time: 00:00:44<br>[...]<br><br>% tail -10 py39-installer-0.6.0_1.log<br>=>> Checking for filesystem violations... done<br>=>> Error: Filesystem touched during build:<br>extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/tomli/__pycache__<br>extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/__pycache__<br>extra: usr/local/lib/python3.9/site-packages/flit_core/__pycache__<br>=>> Cleaning up wrkdir<br>===> Cleaning for py39-installer-0.6.0_1<br>build of devel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb 8 03:11:24 CST 2023<br>build time: 00:00:02<br>!!! build failure encountered !!!<br>% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log<br>% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log | wc -l<br><div> 0</div></div></div>
home |
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMHz58St0=QNVh9BM7pxqsibSJzPFuC-FYAvHhdL=qukBWWhXQ>