From owner-freebsd-python@freebsd.org Tue Jan 31 20:24:29 2017 Return-Path: Delivered-To: freebsd-python@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2EA40CCA478 for ; Tue, 31 Jan 2017 20:24:29 +0000 (UTC) (envelope-from john@saltant.com) Received: from twaddle.saltant.net (twaddle.saltant.net [IPv6:2001:470:8d6f:1001::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E9E177A8 for ; Tue, 31 Jan 2017 20:24:28 +0000 (UTC) (envelope-from john@saltant.com) Received: from dreck.saltant.net (unknown [IPv6:2001:470:8d6f:1001:ee1a:59ff:fe3d:4bb1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by twaddle.saltant.net (Postfix) with ESMTPSA id 6940AFDD3; Tue, 31 Jan 2017 15:24:27 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=saltant.com; s=twaddle; t=1485894267; bh=Djj9ig99T/24febaNocsA0MGqFX0FQwol2zewLU7KV8=; h=Subject:To:References:From:Date:In-Reply-To; b=mgCfTQnYK0DO+DYgJdpra+6/GYTqpqi2jMsiKmRSkpfrXSAX6YQ6QLCfZdnGQhtnQ gDFA7gjSaucvxsaxf9NJGl4OElQIE5dr1B3wHQ3UdiliraPXYJ5HHiZnH7Xw8SI/XX N8h/7eUWFPSQVdP1EMUFXV1qXqpDjnIeZ7yULrQEDKvPAqhedUaxcN+GvkvGQEkG9R 1kQwPJGIFE6q10wDflPP14KaD/hvDIfrtF4tLy9Pw0/B6/YlVpspBBEMury73f9JVO SnxINzztQjfpLXeuT9Tb/HsIV4+plZArtRHt+QfkspdaTWU2s9eMRaGVeW5Ah16r58 jUjPVyKwMGccg== Subject: Re: Need help finding root cause of staging errors related to MANIFEST To: "John W. O'Brien" , FreeBSD Python List References: <13910f98-9041-5a2b-8917-e841df33999b@saltant.com> <5106582f-54dd-503a-ab67-d94d9812d0f9@saltant.com> From: "John W. O'Brien" Organization: Saltant Solutions Message-ID: <6cddcb5f-8c64-d6f4-4a31-a10b4f665166@saltant.com> Date: Tue, 31 Jan 2017 15:24:31 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <5106582f-54dd-503a-ab67-d94d9812d0f9@saltant.com> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="XG6OXAdqLe6EkElaXKqAfLnVGiUMAWOf6" X-BeenThere: freebsd-python@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: FreeBSD-specific Python issues List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jan 2017 20:24:29 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --XG6OXAdqLe6EkElaXKqAfLnVGiUMAWOf6 Content-Type: multipart/mixed; boundary="hdvgfuI8ETTExFvRuMCBiNx3R14lJelrO"; protected-headers="v1" From: "John W. O'Brien" To: "John W. O'Brien" , FreeBSD Python List Message-ID: <6cddcb5f-8c64-d6f4-4a31-a10b4f665166@saltant.com> Subject: Re: Need help finding root cause of staging errors related to MANIFEST References: <13910f98-9041-5a2b-8917-e841df33999b@saltant.com> <5106582f-54dd-503a-ab67-d94d9812d0f9@saltant.com> In-Reply-To: <5106582f-54dd-503a-ab67-d94d9812d0f9@saltant.com> --hdvgfuI8ETTExFvRuMCBiNx3R14lJelrO Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 1/31/17 10:21, John W. O'Brien wrote: > On 1/29/17 21:45, John W. O'Brien wrote: >> At some point between 2017-01-17T07:39:59-05:00 >> 2017-01-23T10:48:15-05:00, some ports started failing to build under >> python3.4 (not 3.5 or 3.6). The failure occurs during the poudriere >> stage phase while processing the MANIFEST.in template. If the manifest= >> contains a graft statement that refers to a non-existent directory, th= e >> build log contains a fatal error where before there was only a warning= =2E > [...] >> The time interval implicates python 3.4.6, and not setuptools 32.1.0. >> I'm having a hard time narrowing it down further than that. >=20 > After reading a bunch of source code and learning about DISTUTILS_DEBUG= , > I obtained the following trace (edited for brevity and clarity). >=20 > =3D=3D=3D=3D > reading manifest template 'MANIFEST.in' > warning: no files found matching 'README.rst' > no previously-included directories found matching 'doc/build' > error: [Errno 2] No such file or directory: 'examples' > Traceback (most recent call last): > File "", line 1, in > File "setup.py", line 696, in > **setuptools_kwargs) > File "/usr/local/lib/python3.4/distutils/core.py", line 148, in setup= > dist.run_commands() > [...] > File > "/usr/local/lib/python3.4/site-packages/setuptools/command/egg_info.py"= , > line 444, in graft > found =3D distutils.filelist.findall(dir) > File "/usr/local/lib/python3.4/distutils/filelist.py", line 258, in > findall > names =3D os.listdir(dir) > FileNotFoundError: [Errno 2] No such file or directory: 'examples' > include MANIFEST.in > include LICENSE > include RELEASE.md > include README.rst > include setup.py > graft doc > prune doc/build > graft examples > *** Error code 1 >=20 > Stop. > make: stopped in /usr/ports/math/py-pandas > =3D=3D=3D=3D >=20 > This points to a patch [0] in python 3.4.6. How does the list think I > should proceed? File a PR with lang/python34, with devel/py-setuptools,= > with one of the upstreams? >=20 > [0] https://hg.python.org/cpython/rev/e82b995d1a5c I compared the implementation of distutils.filelist.findall across the four python versions supported by FreeBSD and found that 2.7 [0] and 3.4 [1] share an implementation, and 3.5 [2] and 3.6 [3] share a different implementation. However: $ for v in 2.7 3.4 3.5 3.6 do python$v -c 'import os; os.listdir("x");' 2>&1 | tail -n1 done OSError: [Errno 2] No such file or directory: 'x' FileNotFoundError: [Errno 2] No such file or directory: 'x' FileNotFoundError: [Errno 2] No such file or directory: 'x' FileNotFoundError: [Errno 2] No such file or directory: 'x' So now we know that the problem only exists in 3.4 because it is the only version (at this moment) where the findall implementation still uses os.listdir but the os.listdir behavior for a nonexistent directory has changed. It's getting harder to track down A) where the change from OSError to FileNotFoundError occurred, and B) where the OSError exception was handled in 2.7.13 and the FileNotFoundError is not handled in 3.4.6. It may not be necessary for the list to take a position on the bug, though. Shall I prepare a work-around in devel/py-setuptools34, submit bug reports to python and/or setuptools, embark upon a campaign to clean non-existent directories from all upstream MANIFESTs, all/none of the above, or something else? [0] https://hg.python.org/cpython/file/v2.7.13/Lib/distutils/filelist.py#l256= [1] https://hg.python.org/cpython/file/v3.4.6/Lib/distutils/filelist.py#l= 245 [2] https://hg.python.org/cpython/file/v3.5.3/Lib/distutils/filelist.py#l= 258 [3] https://hg.python.org/cpython/file/v3.6.0/Lib/distutils/filelist.py#l= 258 --hdvgfuI8ETTExFvRuMCBiNx3R14lJelrO-- --XG6OXAdqLe6EkElaXKqAfLnVGiUMAWOf6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJYkPKDAAoJEGEAJpqmdExHNEQP/2lwRe0pYchJU6LVu3RsjTrX bmNjT7+GTFBpu7rIeJzi45AM76sqaQquRS4zK4RJLAKLVV+v4KRF1n9A+gpNgzdI 4wfd1q1udFL5kHbizkriV6JKGER8GodUO1+ii9dDIB883aWuKKc5oM/cxp5JCdW1 k9apjb/z3fh17KDyxt0p6DGse3wtcPIpKiKDEL86oMCmF+KXVe3ZSBK/wBHlRkfq AxGHcQo5NTRvuxCAhDi85PvkNHcMGaNpG/wkEHI9WnTRoErsN8klsQkB+dglaJpY VfsT3+IvRzWldyz/7+CjnIwuZryNbxqL797IBy5cyDfXSSnDspx1pFlXkwJ+gNRl EY3JtElGiAk3L58MdzJoWbW0mkcPnIU1T3D53v8Yj3XIVxwWuzCd7L2sX+U6vNNG gSAPT8+VwVsW33waY7BclLVpPPaLEYAdIPCTfTTuCDjTl0AIkGNhCkKgDMul8bXm WKdmbYgM4+Lpm1wLiWurVFtVj5lYRa0rLEs/RHUijN3QRPATOWlSsVmKnKtVOuDr Gu+T2AEoVzE8WoTeKMqZRs7JyhqOzLKObnfxVS7eoGrieqbxaObzXlXBGK/r2HWA jDCz/Ezey+GpN4u3kpxA3Lz2MkUM8CPBNdacWDY7/YFOE9/TcS39UrUUQgwx4YEZ /mEofgf6DELYYKNA7PIX =lk5r -----END PGP SIGNATURE----- --XG6OXAdqLe6EkElaXKqAfLnVGiUMAWOf6--