From nobody Tue Feb 7 19:41:16 2023 X-Original-To: dev-commits-ports-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PBD6C5xZWz3ncjh for ; Tue, 7 Feb 2023 19:41:55 +0000 (UTC) (envelope-from sunpoet@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PBD6C5Qvvz4ctr for ; Tue, 7 Feb 2023 19:41:55 +0000 (UTC) (envelope-from sunpoet@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675798915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=pmtYnSLKg9dYvZC2Sdn1NkcYyOCvOf4vdMioGIz39ZE=; b=AgHYZPP+PurSFy3ZzYDIHXhRlTs9XsTteBhJy58F46uu2/BDQELSdv+N3JnL/ZqHmF9QBm XJzlLnCAmxHUnFYlHKSOcfPrgPrHtLzMS/LqN4mW8kXK1xUswGNidpb9n9wOgtqjFcXw0R 03S8cmuRbzgyVo26ODyo4aB58ZhgorfAl51/NEnuArvk0U6KFEvwYmhPx8eCg+FXEp2PZ0 9SkIAOeIphCN1KYTNnvYaqqngEKqKPDOC8PT/Bbcr0rj4iK6vXGSvXBPJePOqErr3bj4IY G4iihYj4BXtGeBb367r07ITwVWeWfQDlCdmr5wJECmmg46tRMPflkUV69gqqaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675798915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=pmtYnSLKg9dYvZC2Sdn1NkcYyOCvOf4vdMioGIz39ZE=; b=P/T1HFOk5lkDAkqg8PD8xVKURKEHdOaYQZkwfTqWiX4F14lQAZ6e9I5eZrJK+T0q4d1mA8 BKpLhEOJ16+Hd/rTk3vxulLUNqELn9X8l/d6V8lwhNNaEIQynRB1CB49AuPjyzLI+fFBH+ fUkXJFYaowW+9W5pc6Q0ZKXHlRcbujFIpyVGKS2oPp/f4pPlkkHVgLMAraV+wLcP9Y0rgR dkNa3ZXLmgg9PrQVvoA3++kVFwsyQWwSbD7QUIwo3lbxqCtcbc/LyM0P7XrNfeEO38dCVn L9uvueCKmcWmL4xMV1aq951nlRhn60NeBRb4L6cqBaa7QmMSwCD6n5NbkL43zw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675798915; a=rsa-sha256; cv=none; b=a4pP1XQasulSWO1rdDTzX+i8FIIBBNBaDyD7IkyOJoF/UN8Rp8Lh9uCbPYPPAnPNA/colC jXLd7t0NFxemMGCKkqcpJtfOUM0dId7QQbvZm4hey1pkDdJm/w/1I7KV2i6UFeM7vJcW/F UZLM3dcWtYDvorHDZRiZymLmLtwBn31AUcQENAKL34Yz26OQfnNOLv7XNtkzjBpst6OKKn 1VIEAJb8uHxs3gKDrsYc7MwTfqJBZKt22cfhHTJyoNleNq4SuhMUPVBvrhFAojSZpCTvnU BV0VgjH7cNXF/kXtJxjor+3+51FbkVhOySwqZ8EIHv8aQl8S+PasN90NquoBOQ== Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: sunpoet) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PBD6C4Lk2zbsX for ; Tue, 7 Feb 2023 19:41:55 +0000 (UTC) (envelope-from sunpoet@freebsd.org) Received: by mail-ed1-f46.google.com with SMTP id eq11so17539999edb.6 for ; Tue, 07 Feb 2023 11:41:55 -0800 (PST) X-Gm-Message-State: AO0yUKU6um9BoiUH4hrmCOWMZq0sCATQxnv5aLHt84h+SWIVkdRdOYnh hY33xWNX1sLo8yu8/Lf4nwt6qg7lot2Vcpf0G4AHEQ== X-Google-Smtp-Source: AK7set+8v9yDUG8mliGx1zR9sUSFtOSiq/oxWuMnE998PmrcmQarhv9Vy9q5fBshmJ0mxh45BHG207gr/WrORXxGt9s= X-Received: by 2002:a50:c31e:0:b0:49d:ec5e:673e with SMTP id a30-20020a50c31e000000b0049dec5e673emr968836edb.6.1675798914722; Tue, 07 Feb 2023 11:41:54 -0800 (PST) List-Id: Commits to the main branch of the FreeBSD ports repository List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-ports-main@freebsd.org X-BeenThere: dev-commits-ports-main@freebsd.org MIME-Version: 1.0 References: <202302051941.315JfQOP071383@gitrepo.freebsd.org> <729ca691-4f25-a094-a6b4-99490674ecd0@freebsd.org> In-Reply-To: <729ca691-4f25-a094-a6b4-99490674ecd0@freebsd.org> From: Po-Chuan Hsieh Date: Wed, 8 Feb 2023 03:41:16 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: git: f5890bd3cbc6 - main - Revert "Mk/Uses/python.mk: Fix USE_PYTHON=pep517: always compile and install bytecode" To: Charlie Li Cc: ports-committers@freebsd.org, dev-commits-ports-all@freebsd.org, dev-commits-ports-main@freebsd.org Content-Type: multipart/alternative; boundary="00000000000007041a05f42157ca" X-ThisMailContainsUnwantedMimeParts: N --00000000000007041a05f42157ca Content-Type: text/plain; charset="UTF-8" On Wed, Feb 8, 2023 at 2:36 AM Charlie Li wrote: > Po-Chuan Hsieh wrote: > > Those issues you mentioned might be a problem in the future but clearly > > not in the short term. > > I'm trying to fix an immediate problem which greatly affects the > > development. > They are problems now, because they impede any proper fixes. We have > been doing quite a few things wrong with Python packaging, and simply > fixing "immediate problems" doesn't work. > > It may not be good from your point of view but it really fixed the > > problem (see below). > > If you have a better solution, I'm fine with it. > > > > > > Instead of compiling and installing bytecode at stage/package > time, > > there is a WIP, review D34739, that compiles and installs > bytecode > > at install time instead, using triggers. > > > > > > Thanks for pointing out a possible solution but it is still WIP. > > > It will be *the* solution, because bytecode is not meant to be packaged, > but only compiled at install or run-time at user's discretion. The only > reason why bytecode was packaged in operating system-level package > distributions is how most such systems lack a trigger function that we > now have. > When Python packaging tooling is the final arbiter, bytecode is compiled > only after the installation process completes if the user does not > disable it. In our case, pkg(8) is the final arbiter, so such tasking > falls to pkg(8)'s triggers and not during stage/packaging. > > > > The aforementioned build_fs_violations will be investigated. > > > > > > You could reproduce this problem as follows. > > 1. Revert d6583d67b69dbf690e6f102e1aec799cb7fd564a (change > > devel/py-importlib-metadata back to USE_PYTHON=pep517 which is the > > background) > > 2. Build a port with USE_PYTHON=pep517. I choose devel/py-pyls-black > here. > > Now it failed earlier with devel/py-click. > > You could expect more ports to be failed/skipped while building hundreds > > of ports in my case. > > > Can't reproduce with a proper PEP-517 bootstrap tooling, which most > importantly does not have setuptools in the environment at all. If > anything, your build failure only demonstrates just how "problematic, > fragile and unreliable" compiling and installing bytecode at > stage/package time is. > The problem exists. I'm not insisting on putting bytecode in the package. But we need a fix or hack before the solution (trigger) lands. Let me explain it in more detail. While building devel/py-installer, PEP517_BUILD_CMD executes flit_core.wheel and the corresponding bytecodes are generated in the __pycache__ directories. The fs outside STAGEDIR is being changed. It makes build_fs_violation failure in poudriere. There is nothing about setuptools. I wonder if you really tested it. This is the up-to-date example (a fresh checkout). % git st devel/py-flit-core devel/py-installer On branch main Your branch is up to date with 'freebsd/main'. nothing to commit, working tree clean % sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core devel/py-installer [...] [00:02:11] [01] [00:00:00] Building lang/python39 | python39-3.9.16 [00:03:13] [01] [00:01:02] Finished lang/python39 | python39-3.9.16: Success [00:03:14] [01] [00:00:00] Building devel/py-flit-core@py39 | py39-flit-core-3.8.0_1 [00:03:17] [01] [00:00:03] Finished devel/py-flit-core@py39 | py39-flit-core-3.8.0_1: Success [00:03:17] [01] [00:00:00] Building devel/py-installer@py39 | py39-installer-0.6.0_1 [00:03:21] [01] [00:00:04] Finished devel/py-installer@py39 | py39-installer-0.6.0_1: Failed: build_fs_violation [...] [00:00:44] Built ports: devel/py-flit-core [00:00:44] Failed ports: devel/py-installer:build_fs_violation [12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2 Built: 1 Failed: 1 Skipped: 0 Ignored: 0 Tobuild: 0 Time: 00:00:44 [...] % tail -10 py39-installer-0.6.0_1.log =>> Checking for filesystem violations... done =>> Error: Filesystem touched during build: extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/tomli/__pycache__ extra: usr/local/lib/python3.9/site-packages/flit_core/vendor/__pycache__ extra: usr/local/lib/python3.9/site-packages/flit_core/__pycache__ =>> Cleaning up wrkdir ===> Cleaning for py39-installer-0.6.0_1 build of devel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb 8 03:11:24 CST 2023 build time: 00:00:02 !!! build failure encountered !!! % grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log % grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log | wc -l 0 --00000000000007041a05f42157ca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Wed, Feb 8, 2023 at 2:36 AM Charlie Li= <vishwin@freeb= sd.org> wrote:
Po-Chuan Hsieh wrote:
> Those issues you mentioned might be a problem in the=C2=A0future but c= learly
> not in the=C2=A0short term.
> I'm trying to fix an=C2=A0immediate problem which greatly affects = the
> development.
They are problems now, because they impede any proper fixes. We have
been doing quite a few things wrong with Python packaging, and simply
fixing "immediate problems" doesn't work.
> It may not be good from=C2=A0your point of view but it=C2=A0really fix= ed the
> problem (see below).
> If you have a better solution, I'm fine with it.
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Instead of compiling and installing = bytecode at stage/package time,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 there is a WIP, review D34739, that = compiles and installs bytecode
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at install time instead, using trigg= ers.
>
>
> Thanks for pointing out a possible solution but it is still WIP.
>
It will be *the* solution, because bytecode is not meant to be packaged, but only compiled at install or run-time at user's discretion. The only=
reason why bytecode was packaged in operating system-level package
distributions is how most such systems lack a trigger function that we
now have.=C2=A0

When Python packaging tooling is the final arbiter, bytecode is compiled only after the installation process completes if the user does not
disable it. In our case, pkg(8) is the final arbiter, so such tasking
falls to pkg(8)'s triggers and not during stage/packaging.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 The aforementioned build_fs_violatio= ns will be investigated.
>
>
> You could reproduce this problem as follows.
> 1. Revert=C2=A0d6583d67b69dbf690e6f102e1aec799cb7fd564a (change
> devel/py-importlib-metadata back to USE_PYTHON=3Dpep517 which is the <= br> > background)
> 2. Build a port with USE_PYTHON=3Dpep517. I choose devel/py-pyls-black= here.
> Now it failed earlier with devel/py-click.
> You could expect more ports to be failed/skipped while building hundre= ds
> of ports in my case.
>
Can't reproduce with a proper PEP-517 bootstrap tooling, which most importantly does not have setuptools in the environment at all. If
anything, your build failure only demonstrates just how "problematic, =
fragile and unreliable" compiling and installing bytecode at
stage/package time is.

The problem exis= ts.
I'm not insisting=C2=A0on putting=C2=A0bytecode in t= he package.
But we need a fix or hack before the solution (trigge= r) lands.

Let me explain it in more detail.
While building devel/py-installer, PEP517_BUILD_CMD executes= flit_core.wheel and the corresponding bytecodes are generated in the __pyc= ache__ directories.
The fs outside STAGEDIR is being changed.=
It makes build_fs_violation failure in poudriere.
There is nothing about setuptools.

I wonder= if you really tested it.
This is the=C2=A0up-to-date example (a = fresh checkout).

% git st devel/py-flit-core= devel/py-installer
On branch main
Your branch is up to date wi= th 'freebsd/main'.

nothing to commit, working tree clean
= % sudo poudriere bulk -j 12 -p testing -r -t devel/py-flit-core devel/py-in= staller
[...]
[00:02:11] [01] [00:00:00] Building lang/python39 | pyt= hon39-3.9.16
[00:03:13] [01] [00:01:02] Finished lang/python39 | python3= 9-3.9.16: Success
[00:03:14] [01] [00:00:00] Building devel/py-flit-core= @py39 | py39-flit-core-3.8.0_1
[00:03:17] [01] [00:00:03] Finished devel= /py-flit-core@py39 | py39-flit-core-3.8.0_1: Success
[00:03:17] [01] [00= :00:00] Building devel/py-installer@py39 | py39-installer-0.6.0_1
[00:03= :21] [01] [00:00:04] Finished devel/py-installer@py39 | py39-installer-0.6.= 0_1: Failed: build_fs_violation
[...]
[00:00:44] Built ports: devel/p= y-flit-core
[00:00:44] Failed ports: devel/py-installer:build_fs_violati= on
[12-default] [2023-02-08_02h44m41s] [committing:] Queued: 2 =C2=A0Bui= lt: 1 =C2=A0Failed: 1 =C2=A0Skipped: 0 =C2=A0Ignored: 0 =C2=A0Tobuild: 0 = =C2=A0 Time: 00:00:44
[...]

% tail -10 py39-installer-0.6.0_1.log=
=3D>> Checking for filesystem violations... done
=3D>> E= rror: Filesystem touched during build:
extra: usr/local/lib/python3.9/si= te-packages/flit_core/vendor/tomli/__pycache__
extra: usr/local/lib/pyth= on3.9/site-packages/flit_core/vendor/__pycache__
extra: usr/local/lib/py= thon3.9/site-packages/flit_core/__pycache__
=3D>> Cleaning up wrkd= ir
=3D=3D=3D> =C2=A0Cleaning for py39-installer-0.6.0_1
build of d= evel/py-installer | py39-installer-0.6.0_1 ended at Wed Feb =C2=A08 03:11:2= 4 CST 2023
build time: 00:00:02
!!! build failure encountered !!!
= % grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log
= % grep setuptools py39-flit-core-3.8.0_1.log py39-installer-0.6.0_1.log | w= c -l
=C2=A0 =C2=A0 =C2=A0 =C2=A00
--00000000000007041a05f42157ca--