From owner-freebsd-toolchain@freebsd.org Mon Jun 18 23:29:16 2018 Return-Path: Delivered-To: freebsd-toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57AD910057F6; Mon, 18 Jun 2018 23:29:16 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA39479868; Mon, 18 Jun 2018 23:29:15 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id B63DA3D93; Mon, 18 Jun 2018 23:29:15 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id A1290A564; Mon, 18 Jun 2018 23:29:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id GZcIZP0abfkK; Mon, 18 Jun 2018 23:29:07 +0000 (UTC) To: Li-Wen Hsu , Mark Millard DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com BF589A55F Cc: FreeBSD Current , FreeBSD Toolchain References: <74EAD684-0E0B-453A-B746-156777CF604A@yahoo.com> <1884103f-d1fb-aca6-2edd-062e11d05617@FreeBSD.org> From: Bryan Drewery Openpgp: id=F9173CB2C3AAEA7A5C8A1F0935D771BB6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Autocrypt: addr=bdrewery@FreeBSD.org; prefer-encrypt=mutual; keydata= xsBNBFJphmsBCADiFgmS4bIzwZijrS31SjEMzg+n5zNellgM+HkShwehpqCiyhXdWrvH6dTZ a6u50pbUIX7doTR7W7PQHCjCTqtpwvcj0eulZva+iHFp+XrbgSFHn+VVXgkYP2MFySyZRFab D2qqzJBEJofhpv4HvY6uQI5K99pMqKr1Z/lHqsijYYu4RH2OfwB5PinId7xeldzWEonVoCr+ rfxzO/UrgA6v/3layGZcKNHFjmc3NqoN1DXtdaEHqtjIozzbndVkH6lkFvIpIrI6i5ox8pwp VxsxLCr/4Musd5CWgHiet5kSw2SzNeA8FbxdLYCpXNVu+uBACEbCUP+CSNy3NVfEUxsBABEB AAHNJEJyeWFuIERyZXdlcnkgPGJkcmV3ZXJ5QEZyZWVCU0Qub3JnPsLAgAQTAQoAKgIbAwUL CQgHAwUVCgkICwUWAwIBAAIeAQIXgAIZAQUCWujOIgUJCmB7NwAKCRA113G7bkaXz/xpB/9b /UWIPbieY1IeIuHF2pyYPE7Hytkh3HVsxMA0F5Ma2AYQsXZZeKNKWrF7RPyDyDwUklLHJkhm k3EfClBbHxf08kMIm1vWCJRtgxic9knY/bzYGiWMpHjg3cSd1XfrYH1autYqTZAjDwIkgOjU dR//Tbn4V36sY7y2jz+kdMVWvK53U32aZqiwBbCn4DPe1wSZcUs17mV/0uZdIoGdj74B1orN A/0py5vHYo6HcbBNoaR8pKRLf5VZNRsxqGIMhTucx4SJWcHpuRBWYyvJSFzwvxdK4ZD4Yqoc kFGPVtOXktVMai9exrLvP3G77fKMu8DI6j4QRU4wCesnHuIfRPFuzsBNBFJphmsBCACiVFPf kNfaFtUSuY0395ueo/rMyHPGPQ2iwvERFCpeFGSQSgagpenNHLpFQKTg/dl6FOoST5tqyxMq fyHGHDzzU51bvA/IfaGoNi/BIhTe/toZNMRvpcI3PLjiGcnJnuwCCbAVOAGdb+t5cZtpNdOI cKYmrYG3u9RiBpe6dTF+qLrD/8Bs1wjhduQ8fcNNgnkXu8xDH4ZxY0lIc3QgvYWp9vimlQe6 iKjUd2/DX28ETZcD5h6pYV331KMPTrEI0p0yvFijUZce8c1XHFyL1j9sBAha5qpszJl6Uq5i LolhKRcGfcdmtD72vHQjUYglUyudSJUVyo2gMYjdbiFKzJulABEBAAHCwGUEGAEKAA8CGwwF AlrozigFCQpgez0ACgkQNddxu25Gl8+m5Af/R3VEdxNMAcDIes9ADhQyofj20SPV3eCJ3HYR OebTSuNdOudGt4AAyA8Ks94u9hiIp5IGsc6RDsT9W7O2vgXhd6eV3eiY5Oif5xLIYrIDVu1Y 1GyRxRrPEn/QOqDN6uFZCPwK1aOapGcYCrO9lB0gMuTVfgHanU61rgC9tMX0OoAOyRd+V3/M 8lDNhjJdF/IpO3SdYzKfkwduy4qamw4Gphcx/RfYQvYLq/eDkP8d50PphWdboqWBwNRHayro W/07OGzfxM5fJ5mBsXPQcO2QcRjkyHf6xCM6Hi1qQL4OnXMNE/ZTX0lnOj1/pH93TlzSHZMP TaiiA/MBD3vGsXBmBg== Organization: FreeBSD Subject: Re: A head buildworld race visible in the ci.freebsd.org build history Message-ID: <33a43aac-231f-6158-1de4-f5dbfaf195df@FreeBSD.org> Date: Mon, 18 Jun 2018 16:29:05 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="i3QlIgdhlde0b31PmQhO90vQjY4MPnZCT" X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 23:29:16 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --i3QlIgdhlde0b31PmQhO90vQjY4MPnZCT Content-Type: multipart/mixed; boundary="DAW6nsCxBca1w2EzimZDNtVm0TBjJ6W4p"; protected-headers="v1" From: Bryan Drewery To: Li-Wen Hsu , Mark Millard Cc: FreeBSD Current , FreeBSD Toolchain Message-ID: <33a43aac-231f-6158-1de4-f5dbfaf195df@FreeBSD.org> Subject: Re: A head buildworld race visible in the ci.freebsd.org build history References: <74EAD684-0E0B-453A-B746-156777CF604A@yahoo.com> <1884103f-d1fb-aca6-2edd-062e11d05617@FreeBSD.org> In-Reply-To: --DAW6nsCxBca1w2EzimZDNtVm0TBjJ6W4p Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 6/18/2018 3:27 PM, Li-Wen Hsu wrote: > On Mon, Jun 18, 2018 at 5:04 PM Mark Millard via freebsd-toolchain > wrote: >> >> On 2018-Jun-18, at 12:42 PM, Bryan Drewery w= rote: >> >>> On 6/15/2018 10:55 PM, Mark Millard wrote: >>>> In watching ci.freebsd.org builds I've seen a notable >>>> number of one time failures, such as (example from >>>> powerpc64): >>>> >>>> --- all_subdir_lib/libufs --- >>>> ranlib -D libufs.a >>>> ranlib: fatal: Failed to open 'libufs.a' >>>> *** [libufs.a] Error code 70 >>>> >>>> where the next build works despite the change being >>>> irrelevant to whatever ranlib complained about. >>>> >>>> Other builds failed similarly: >>>> >>>> --- all_subdir_lib/libbsm --- >>>> ranlib -D libbsm_p.a >>>> ranlib: fatal: Failed to open 'libbsm_p.a' >>>> *** [libbsm_p.a] Error code 70 >>>> >>>> and: >>>> >>>> --- kerberos5/lib__L --- >>>> ranlib -D libgssapi_spnego_p.a >>>> --- libgssapi_spnego.a --- >>>> ranlib -D libgssapi_spnego.a >>>> --- libgssapi_spnego_p.a --- >>>> ranlib: fatal: Failed to open 'libgssapi_spnego_p.a' >>>> *** [libgssapi_spnego_p.a] Error code 70 >>>> >>>> and so on. >>>> >>>> >>>> It is not limited to powerpc64. For example, for aarch64 >>>> there are: >>>> >>>> --- libpam_exec.a --- >>>> building static pam_exec library >>>> ar -crD libpam_exec.a `NM=3D'nm' NMFLAGS=3D'' lorder pam_exec.o | = tsort -q` >>>> ranlib -D libpam_exec.a >>>> ranlib: fatal: Failed to open 'libpam_exec.a' >>>> *** [libpam_exec.a] Error code 70 >>>> >>>> and: >>>> >>>> --- all_subdir_lib/libusb --- >>>> ranlib -D libusb.a >>>> ranlib: fatal: Failed to open 'libusb.a' >>>> *** [libusb.a] Error code 70 >>>> >>>> and: >>>> >>>> --- all_subdir_lib/libbsnmp --- >>>> ranlib: fatal: Failed to open 'libbsnmp.a' >>>> --- all_subdir_lib/ncurses --- >>>> --- all_subdir_lib/ncurses/panelw --- >>>> --- panel.pico --- >>>> --- all_subdir_lib/libbsnmp --- >>>> *** [libbsnmp.a] Error code 70 >>>> >>>> >>>> Even amd64 gets such: >>>> >>>> --- libpcap.a --- >>>> ranlib -D libpcap.a >>>> ranlib: fatal: Failed to open 'libpcap.a' >>>> *** [libpcap.a] Error code 70 >>>> >>>> and: >>>> >>>> >>>> --- libkafs5.a --- >>>> ranlib: fatal: Failed to open 'libkafs5.a' >>>> --- libkafs5_p.a --- >>>> ranlib: fatal: Failed to open 'libkafs5_p.a' >>>> --- cddl/lib__L --- >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua/lbaselib= =2Ec:60:26: note: include the header or explicitly provide a de= claration for 'toupper' >>>> --- kerberos5/lib__L --- >>>> *** [libkafs5_p.a] Error code 70 >>>> >>>> make[5]: stopped in /usr/src/kerberos5/lib/libkafs5 >>>> --- libkafs5.a --- >>>> *** [libkafs5.a] Error code 70 >>>> >>>> and: >>>> >>>> >>>> --- lib__L --- >>>> ranlib -D libclang_rt.asan_cxx-i386.a >>>> ranlib: fatal: Failed to open 'libclang_rt.asan_cxx-i386.a' >>>> *** [libclang_rt.asan_cxx-i386.a] Error code 70 >>>> >>>> >>>> (Notice the variability in what .a the ranlib's fail for.) >>>> >>>> >>>> >>>> >>>> >>> >>> >>> I looked at this a few days ago and don't believe it's actually a bui= ld >>> race. I think there is something wrong with the ar/ranlib on that sys= tem >>> or something else. I've found no evidence of concurrent building of t= he >>> .a files in question. >> >> >> Looking at a bunch of the failures, spanning multiple >> FreeBSD-head-*-build types of builds, I see only: >> >> NODE_LABELS bhyve_host butler1.nyi.freebsd.org jailer jailer_fast >> NODE_NAME butler1.nyi.freebsd.org >> >> for the failures that I looked at. >> >> So your "on that system" might well be correct. >=20 > Thanks for the insight, the build is done in a 11.1-R jail on a > -CURRENT host. butler1.nyi is running r333388 (as a canary) while > other builders are mostly running r328278. I upgraded few others and > it seems can reproduce the issue, and now I downgraded all the build > slaves to r328278 before we find the root cause. >=20 The error is coming from libarchive which had a change between those revisions: > -----------------------------------------------------------------------= - > r328332 | mm | 2018-01-24 06:24:17 -0800 (Wed, 24 Jan 2018) | 14 lines >=20 > MFV r328323,328324: > Sync libarchive with vendor. >=20 > Relevant vendor changes: > PR #893: delete dead ppmd7 alloc callbacks > PR #904: Fix archive freeing bug in bsdcat > PR #961: Fix ZIP format names > PR #962: Don't modify attributes for existing directories > when ARCHIVE_EXTRACT_NO_OVERWRITE is set > PR #964: Fix -Werror=3Dimplicit-fallthrough=3D for GCC 7 > PR #970: zip: Allow backslash as path separator >=20 > MFC after: 1 week >=20 > -----------------------------------------------------------------------= - Nothing obvious stands out in the change to me though from a brief look. --=20 Regards, Bryan Drewery --DAW6nsCxBca1w2EzimZDNtVm0TBjJ6W4p-- --i3QlIgdhlde0b31PmQhO90vQjY4MPnZCT Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJbKEBBAAoJEDXXcbtuRpfPmD8H/i03v1lLcO6wmdDY0EJ9BZzz jhzqJukgt5kfgoj4N7yX/XsErGtwQqgbSDa5oQ/Hfbhwlsl+McpSiNyLgiKmz35Y u0LPuGQRAmcygyScGKKIlL8rZKdw77G7Ic4sIuXJcXtaJeQNm7iq0b7od0vCP9ZA U8XQgfrBNVgkvPIBdkQyKcIqAN+XCG5+0woUzKjQgZN5YtIw5qqRzOtCdx/qy/C/ S26ZR8mz9JTWG9BKqL2TUoo+++j0+Uv9+LDmqdbMi93A9LFKGSDzD8IEl9eOroXV pBy79ecl2lSKUadAG/BXtyT/34rub6mrTRUwFAPrWEw21ALWiw3kkQbGXocTFRw= =UMfb -----END PGP SIGNATURE----- --i3QlIgdhlde0b31PmQhO90vQjY4MPnZCT--