Date: Wed, 8 Feb 2023 03:41:16 +0800 From: Po-Chuan Hsieh <sunpoet@freebsd.org> To: Charlie Li <vishwin@freebsd.org> Cc: ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Subject: Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode" Message-ID: <CAMHz58St0=QNVh9BM7pxqsibSJzPFuC-FYAvHhdL=qukBWWhXQ@mail.gmail.com> In-Reply-To: <729ca691-4f25-a094-a6b4-99490674ecd0@freebsd.org> References: <202302051941.315JfQOP071383@gitrepo.freebsd.org> <CAMHz58QuYSdSi0J93aLoBbjfR3ncizOjFcYyXizUZ%2BP0hji_pg@mail.gmail.com> <729ca691-4f25-a094-a6b4-99490674ecd0@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000007041a05f42157ca
Content-Type: text/plain; charset="UTF-8"
On Wed, Feb 8, 2023 at 2:36 AM Charlie Li <vishwin@freebsd.org> wrote:
> Po-Chuan Hsieh wrote:
> > Those issues you mentioned might be a problem in the future but clearly
> > not in the short term.
> > I'm trying to fix an immediate problem which greatly affects the
> > development.
> They are problems now, because they impede any proper fixes. We have
> been doing quite a few things wrong with Python packaging, and simply
> fixing "immediate problems" doesn't work.
> > It may not be good from your point of view but it really fixed the
> > problem (see below).
> > If you have a better solution, I'm fine with it.
> >
> >
> > Instead of compiling and installing bytecode at stage/package
> time,
> > there is a WIP, review D34739, that compiles and installs
> bytecode
> > at install time instead, using triggers.
> >
> >
> > Thanks for pointing out a possible solution but it is still WIP.
> >
> It will be *the* solution, because bytecode is not meant to be packaged,
> but only compiled at install or run-time at user's discretion. The only
> reason why bytecode was packaged in operating system-level package
> distributions is how most such systems lack a trigger function that we
> now have.
> When Python packaging tooling is the final arbiter, bytecode is compiled
> only after the installation process completes if the user does not
> disable it. In our case, pkg(8) is the final arbiter, so such tasking
> falls to pkg(8)'s triggers and not during stage/packaging.
> >
> > The aforementioned build_fs_violations will be investigated.
> >
> >
> > You could reproduce this problem as follows.
> > 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change
> > devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the
> > background)
> > 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black
> here.
> > Now it failed earlier with devel/py-click.
> > You could expect more ports to be failed/skipped while building hundreds
> > of ports in my case.
> >
> Can't reproduce with a proper PEP-517 bootstrap tooling, which most
> importantly does not have setuptools in the environment at all. If
> anything, your build failure only demonstrates just how "problematic,
> fragile and unreliable" compiling and installing bytecode at
> stage/package time is.
>
The problem exists.
I'm not insisting on putting bytecode in the package.
But we need a fix or hack before the solution (trigger) lands.
Let me explain it in more detail.
While building devel/py-installer, PEP517_BUILD_CMD executes
flit_core.wheel and the corresponding bytecodes are generated in the
__pycache__ directories.
The fs outside STAGEDIR is being changed.
It makes build_fs_violation failure in poudriere.
There is nothing about setuptools.
I wonder if you really tested it.
This is the up-to-date example (a fresh checkout).
% git st devel/py-flit-core devel/py-installer
On branch main
Your branch is up to date with 'freebsd/main'.
nothing to commit, working tree clean
% sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core
devel/py-installer
[...]
[00:02:11] [01] [00:00:00] Building lang/python39 | python39-3.9.16
[00:03:13] [01] [00:01:02] Finished lang/python39 | python39-3.9.16: Success
[00:03:14] [01] [00:00:00] Building devel/py-flit-core@py39 |
py39-flit-core-3.8.0_1
[00:03:17] [01] [00:00:03] Finished devel/py-flit-core@py39 |
py39-flit-core-3.8.0_1: Success
[00:03:17] [01] [00:00:00] Building devel/py-installer@py39 |
py39-installer-0.6.0_1
[00:03:21] [01] [00:00:04] Finished devel/py-installer@py39 |
py39-installer-0.6.0_1: Failed: build_fs_violation
[...]
[00:00:44] Built ports: devel/py-flit-core
[00:00:44] Failed ports: devel/py-installer:build_fs_violation
[12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2 Built: 1
Failed: 1 Skipped: 0 Ignored: 0 Tobuild: 0 Time: 00:00:44
[...]
% tail -10 py39-installer-0.6.0_1.log
=>> Checking for filesystem violations... done
=>> Error: Filesystem touched during build:
extra:
usr/local/lib/python3.9/site-packages/flit_core/vendor/tomli/__pycache__
extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/__pycache__
extra: usr/local/lib/python3.9/site-packages/flit_core/__pycache__
=>> Cleaning up wrkdir
===> Cleaning for py39-installer-0.6.0_1
build of devel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb 8
03:11:24 CST 2023
build time: 00:00:02
!!! build failure encountered !!!
% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log
% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log |
wc -l
0
--00000000000007041a05f42157ca
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr">On Wed, Feb 8, 2023 at 2:36 AM Charlie Li=
<<a href=3D"mailto:vishwin@freebsd.org" target=3D"_blank">vishwin@freeb=
sd.org</a>> wrote:<br></div><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">Po-Chuan Hsieh wrote:<br>
> Those issues you mentioned might be a problem in the=C2=A0future but c=
learly <br>
> not in the=C2=A0short term.<br>
> I'm trying to fix an=C2=A0immediate problem which greatly affects =
the <br>
> development.<br>
They are problems now, because they impede any proper fixes. We have <br>
been doing quite a few things wrong with Python packaging, and simply <br>
fixing "immediate problems" doesn't work.<br>
> It may not be good from=C2=A0your point of view but it=C2=A0really fix=
ed the <br>
> problem (see below).<br>
> If you have a better solution, I'm fine with it.<br>
> <br>
> <br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Instead of compiling and installing =
bytecode at stage/package time,<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 there is a WIP, review D34739, that =
compiles and installs bytecode<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at install time instead, using trigg=
ers.<br>
> <br>
> <br>
> Thanks for pointing out a possible solution but it is still WIP.<br>
> <br>
It will be *the* solution, because bytecode is not meant to be packaged, <b=
r>
but only compiled at install or run-time at user's discretion. The only=
<br>
reason why bytecode was packaged in operating system-level package <br>
distributions is how most such systems lack a trigger function that we <br>
now have.=C2=A0</blockquote><blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x">
<br>
When Python packaging tooling is the final arbiter, bytecode is compiled <b=
r>
only after the installation process completes if the user does not <br>
disable it. In our case, pkg(8) is the final arbiter, so such tasking <br>
falls to pkg(8)'s triggers and not during stage/packaging.<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 The aforementioned build_fs_violatio=
ns will be investigated.<br>
> <br>
> <br>
> You could reproduce this problem as follows.<br>
> 1. Revert=C2=A0d6583d67b69dbf690e6f102e1aec799cb7fd564a (change <br>
> devel/py-importlib-metadata back to USE_PYTHON=3Dpep517 which is the <=
br>
> background)<br>
> 2. Build a port with USE_PYTHON=3Dpep517. I choose devel/py-pyls-black=
here.<br>
> Now it failed earlier with devel/py-click.<br>
> You could expect more ports to be failed/skipped while building hundre=
ds <br>
> of ports in my case.<br>
> <br>
Can't reproduce with a proper PEP-517 bootstrap tooling, which most <br=
>
importantly does not have setuptools in the environment at all. If <br>
anything, your build failure only demonstrates just how "problematic, =
<br>
fragile and unreliable" compiling and installing bytecode at <br>
stage/package time is.<br></blockquote><div><br></div><div>The problem exis=
ts.</div><div><div>I'm not insisting=C2=A0on putting=C2=A0bytecode in t=
he package.</div><div>But we need a fix or hack before the solution (trigge=
r) lands.</div><div><br></div><div><div>Let me explain it in more detail.</=
div></div><div>While building devel/py-installer, PEP517_BUILD_CMD executes=
flit_core.wheel and the corresponding bytecodes are generated in the __pyc=
ache__ directories.<br></div><div>The fs outside STAGEDIR is being changed.=
</div><div>It makes build_fs_violation failure in poudriere.</div></div><di=
v>There is nothing about setuptools.</div><div><br></div><div><div>I wonder=
if you really tested it.</div><div>This is the=C2=A0up-to-date example (a =
fresh checkout).</div></div><div><br></div><div>% git st devel/py-flit-core=
devel/py-installer<br></div>On branch main<br>Your branch is up to date wi=
th 'freebsd/main'.<br><br>nothing to commit, working tree clean<br>=
% sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core devel/py-in=
staller<br>[...]<br>[00:02:11] [01] [00:00:00] Building lang/python39 | pyt=
hon39-3.9.16<br>[00:03:13] [01] [00:01:02] Finished lang/python39 | python3=
9-3.9.16: Success<br>[00:03:14] [01] [00:00:00] Building devel/py-flit-core=
@py39 | py39-flit-core-3.8.0_1<br>[00:03:17] [01] [00:00:03] Finished devel=
/py-flit-core@py39 | py39-flit-core-3.8.0_1: Success<br>[00:03:17] [01] [00=
:00:00] Building devel/py-installer@py39 | py39-installer-0.6.0_1<br>[00:03=
:21] [01] [00:00:04] Finished devel/py-installer@py39 | py39-installer-0.6.=
0_1: Failed: build_fs_violation<br>[...]<br>[00:00:44] Built ports: devel/p=
y-flit-core<br>[00:00:44] Failed ports: devel/py-installer:build_fs_violati=
on<br>[12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2 =C2=A0Bui=
lt: 1 =C2=A0Failed: 1 =C2=A0Skipped: 0 =C2=A0Ignored: 0 =C2=A0Tobuild: 0 =
=C2=A0 Time: 00:00:44<br>[...]<br><br>% tail -10 py39-installer-0.6.0_1.log=
<br>=3D>> Checking for filesystem violations... done<br>=3D>> E=
rror: Filesystem touched during build:<br>extra: usr/local/lib/python3.9/si=
te-packages/flit_core/vendor/tomli/__pycache__<br>extra: usr/local/lib/pyth=
on3.9/site-packages/flit_core/vendor/__pycache__<br>extra: usr/local/lib/py=
thon3.9/site-packages/flit_core/__pycache__<br>=3D>> Cleaning up wrkd=
ir<br>=3D=3D=3D> =C2=A0Cleaning for py39-installer-0.6.0_1<br>build of d=
evel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb =C2=A08 03:11:2=
4 CST 2023<br>build time: 00:00:02<br>!!! build failure encountered !!!<br>=
% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log<br>=
% grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log | w=
c -l<br><div>=C2=A0 =C2=A0 =C2=A0 =C2=A00</div></div></div>
--00000000000007041a05f42157ca--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAMHz58St0=QNVh9BM7pxqsibSJzPFuC-FYAvHhdL=qukBWWhXQ>