Date: Tue, 31 Jan 2017 15:24:31 -0500 From: "John W. O'Brien" <john@saltant.com> To: "John W. O'Brien" <john@saltant.com>, FreeBSD Python List <freebsd-python@freebsd.org> Subject: Re: Need help finding root cause of staging errors related to MANIFEST Message-ID: <6cddcb5f-8c64-d6f4-4a31-a10b4f665166@saltant.com> In-Reply-To: <5106582f-54dd-503a-ab67-d94d9812d0f9@saltant.com> References: <13910f98-9041-5a2b-8917-e841df33999b@saltant.com> <5106582f-54dd-503a-ab67-d94d9812d0f9@saltant.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --XG6OXAdqLe6EkElaXKqAfLnVGiUMAWOf6 Content-Type: multipart/mixed; boundary="hdvgfuI8ETTExFvRuMCBiNx3R14lJelrO"; protected-headers="v1" From: "John W. O'Brien" <john@saltant.com> To: "John W. O'Brien" <john@saltant.com>, FreeBSD Python List <freebsd-python@freebsd.org> Message-ID: <6cddcb5f-8c64-d6f4-4a31-a10b4f665166@saltant.com> Subject: Re: Need help finding root cause of staging errors related to MANIFEST References: <13910f98-9041-5a2b-8917-e841df33999b@saltant.com> <5106582f-54dd-503a-ab67-d94d9812d0f9@saltant.com> In-Reply-To: <5106582f-54dd-503a-ab67-d94d9812d0f9@saltant.com> --hdvgfuI8ETTExFvRuMCBiNx3R14lJelrO Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 1/31/17 10:21, John W. O'Brien wrote: > On 1/29/17 21:45, John W. O'Brien wrote: >> At some point between 2017-01-17T07:39:59-05:00 >> 2017-01-23T10:48:15-05:00, some ports started failing to build under >> python3.4 (not 3.5 or 3.6). The failure occurs during the poudriere >> stage phase while processing the MANIFEST.in template. If the manifest= >> contains a graft statement that refers to a non-existent directory, th= e >> build log contains a fatal error where before there was only a warning= =2E > [...] >> The time interval implicates python 3.4.6, and not setuptools 32.1.0. >> I'm having a hard time narrowing it down further than that. >=20 > After reading a bunch of source code and learning about DISTUTILS_DEBUG= , > I obtained the following trace (edited for brevity and clarity). >=20 > =3D=3D=3D=3D > reading manifest template 'MANIFEST.in' > warning: no files found matching 'README.rst' > no previously-included directories found matching 'doc/build' > error: [Errno 2] No such file or directory: 'examples' > Traceback (most recent call last): > File "<string>", line 1, in <module> > File "setup.py", line 696, in <module> > **setuptools_kwargs) > File "/usr/local/lib/python3.4/distutils/core.py", line 148, in setup= > dist.run_commands() > [...] > File > "/usr/local/lib/python3.4/site-packages/setuptools/command/egg_info.py"= , > line 444, in graft > found =3D distutils.filelist.findall(dir) > File "/usr/local/lib/python3.4/distutils/filelist.py", line 258, in > findall > names =3D os.listdir(dir) > FileNotFoundError: [Errno 2] No such file or directory: 'examples' > include MANIFEST.in > include LICENSE > include RELEASE.md > include README.rst > include setup.py > graft doc > prune doc/build > graft examples > *** Error code 1 >=20 > Stop. > make: stopped in /usr/ports/math/py-pandas > =3D=3D=3D=3D >=20 > This points to a patch [0] in python 3.4.6. How does the list think I > should proceed? File a PR with lang/python34, with devel/py-setuptools,= > with one of the upstreams? >=20 > [0] https://hg.python.org/cpython/rev/e82b995d1a5c I compared the implementation of distutils.filelist.findall across the four python versions supported by FreeBSD and found that 2.7 [0] and 3.4 [1] share an implementation, and 3.5 [2] and 3.6 [3] share a different implementation. However: $ for v in 2.7 3.4 3.5 3.6 do python$v -c 'import os; os.listdir("x");' 2>&1 | tail -n1 done OSError: [Errno 2] No such file or directory: 'x' FileNotFoundError: [Errno 2] No such file or directory: 'x' FileNotFoundError: [Errno 2] No such file or directory: 'x' FileNotFoundError: [Errno 2] No such file or directory: 'x' So now we know that the problem only exists in 3.4 because it is the only version (at this moment) where the findall implementation still uses os.listdir but the os.listdir behavior for a nonexistent directory has changed. It's getting harder to track down A) where the change from OSError to FileNotFoundError occurred, and B) where the OSError exception was handled in 2.7.13 and the FileNotFoundError is not handled in 3.4.6. It may not be necessary for the list to take a position on the bug, though. Shall I prepare a work-around in devel/py-setuptools34, submit bug reports to python and/or setuptools, embark upon a campaign to clean non-existent directories from all upstream MANIFESTs, all/none of the above, or something else? [0] https://hg.python.org/cpython/file/v2.7.13/Lib/distutils/filelist.py#l256= [1] https://hg.python.org/cpython/file/v3.4.6/Lib/distutils/filelist.py#l= 245 [2] https://hg.python.org/cpython/file/v3.5.3/Lib/distutils/filelist.py#l= 258 [3] https://hg.python.org/cpython/file/v3.6.0/Lib/distutils/filelist.py#l= 258 --hdvgfuI8ETTExFvRuMCBiNx3R14lJelrO-- --XG6OXAdqLe6EkElaXKqAfLnVGiUMAWOf6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJYkPKDAAoJEGEAJpqmdExHNEQP/2lwRe0pYchJU6LVu3RsjTrX bmNjT7+GTFBpu7rIeJzi45AM76sqaQquRS4zK4RJLAKLVV+v4KRF1n9A+gpNgzdI 4wfd1q1udFL5kHbizkriV6JKGER8GodUO1+ii9dDIB883aWuKKc5oM/cxp5JCdW1 k9apjb/z3fh17KDyxt0p6DGse3wtcPIpKiKDEL86oMCmF+KXVe3ZSBK/wBHlRkfq AxGHcQo5NTRvuxCAhDi85PvkNHcMGaNpG/wkEHI9WnTRoErsN8klsQkB+dglaJpY VfsT3+IvRzWldyz/7+CjnIwuZryNbxqL797IBy5cyDfXSSnDspx1pFlXkwJ+gNRl EY3JtElGiAk3L58MdzJoWbW0mkcPnIU1T3D53v8Yj3XIVxwWuzCd7L2sX+U6vNNG gSAPT8+VwVsW33waY7BclLVpPPaLEYAdIPCTfTTuCDjTl0AIkGNhCkKgDMul8bXm WKdmbYgM4+Lpm1wLiWurVFtVj5lYRa0rLEs/RHUijN3QRPATOWlSsVmKnKtVOuDr Gu+T2AEoVzE8WoTeKMqZRs7JyhqOzLKObnfxVS7eoGrieqbxaObzXlXBGK/r2HWA jDCz/Ezey+GpN4u3kpxA3Lz2MkUM8CPBNdacWDY7/YFOE9/TcS39UrUUQgwx4YEZ /mEofgf6DELYYKNA7PIX =lk5r -----END PGP SIGNATURE----- --XG6OXAdqLe6EkElaXKqAfLnVGiUMAWOf6--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6cddcb5f-8c64-d6f4-4a31-a10b4f665166>