From owner-freebsd-toolchain@freebsd.org Mon Jun 18 22:33:03 2018 Return-Path: Delivered-To: freebsd-toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 124C910022F9; Mon, 18 Jun 2018 22:33:03 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-wr0-f194.google.com (mail-wr0-f194.google.com [209.85.128.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 93F7777574; Mon, 18 Jun 2018 22:33:02 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-wr0-f194.google.com with SMTP id w10-v6so18449429wrk.9; Mon, 18 Jun 2018 15:33:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=r9Wj0z7HLzWRUnIIO9YK6F3UwbDg6QhGUkO5lX/l7Lk=; b=CQtjKVjG/2yjBPQsu/RFczl1yQ1+g//5by4CJnMY1rpHVknKxdjFKcSUeS9A0K9ZRD G/Xb8d1V5AUN9XKDemBoTgysbzh/TxghOmyJ94o9OVs4lnOx0VlxdmyzltMbfHn1Baws WVSYjZacvmp+j+eLV3va31rbK3dr2mc0ancsuJ7IEDo63GlIufywxK0znyLElxVl6wNh Fw5XKS6miNJqwQhIJFo0/Sw/LUHLeMXKDt4EfNGLAsrcyaPzE85PE+RD7q/Z2u43ztJX op+8uH+AJtKa/idNu/1Z6nWZIA/HdS0J5+orglB+HLDJihyzwqv0AznctRmHd5RKZQbr DvIQ== X-Gm-Message-State: APt69E3FgWIBF5LEllolrbgO2hEKTRVob8JuGRqSzEliKCfNFrQeR0Nk EAgwGWEv07FH6nlK3skbPouEvNZ3SnxB1avCGlk= X-Google-Smtp-Source: ADUXVKIkRjT6pu+maTgIWJiZ3Fv3PC8i0gBxShcdqQN0zyDX8jvTe5uxK2V244bYr7YbEjHGsaq6bHOyAZujULBx48I= X-Received: by 2002:adf:ea44:: with SMTP id j4-v6mr11973928wrn.224.1529360859237; Mon, 18 Jun 2018 15:27:39 -0700 (PDT) MIME-Version: 1.0 References: <74EAD684-0E0B-453A-B746-156777CF604A@yahoo.com> <1884103f-d1fb-aca6-2edd-062e11d05617@FreeBSD.org> In-Reply-To: From: Li-Wen Hsu Date: Mon, 18 Jun 2018 18:27:27 -0400 Message-ID: Subject: Re: A head buildworld race visible in the ci.freebsd.org build history To: Mark Millard Cc: Bryan Drewery , FreeBSD Current , FreeBSD Toolchain Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 22:33:03 -0000 On Mon, Jun 18, 2018 at 5:04 PM Mark Millard via freebsd-toolchain wrote: > > On 2018-Jun-18, at 12:42 PM, Bryan Drewery wrote: > > > On 6/15/2018 10:55 PM, Mark Millard wrote: > >> In watching ci.freebsd.org builds I've seen a notable > >> number of one time failures, such as (example from > >> powerpc64): > >> > >> --- all_subdir_lib/libufs --- > >> ranlib -D libufs.a > >> ranlib: fatal: Failed to open 'libufs.a' > >> *** [libufs.a] Error code 70 > >> > >> where the next build works despite the change being > >> irrelevant to whatever ranlib complained about. > >> > >> Other builds failed similarly: > >> > >> --- all_subdir_lib/libbsm --- > >> ranlib -D libbsm_p.a > >> ranlib: fatal: Failed to open 'libbsm_p.a' > >> *** [libbsm_p.a] Error code 70 > >> > >> and: > >> > >> --- kerberos5/lib__L --- > >> ranlib -D libgssapi_spnego_p.a > >> --- libgssapi_spnego.a --- > >> ranlib -D libgssapi_spnego.a > >> --- libgssapi_spnego_p.a --- > >> ranlib: fatal: Failed to open 'libgssapi_spnego_p.a' > >> *** [libgssapi_spnego_p.a] Error code 70 > >> > >> and so on. > >> > >> > >> It is not limited to powerpc64. For example, for aarch64 > >> there are: > >> > >> --- libpam_exec.a --- > >> building static pam_exec library > >> ar -crD libpam_exec.a `NM='nm' NMFLAGS='' lorder pam_exec.o | tsort -q` > >> ranlib -D libpam_exec.a > >> ranlib: fatal: Failed to open 'libpam_exec.a' > >> *** [libpam_exec.a] Error code 70 > >> > >> and: > >> > >> --- all_subdir_lib/libusb --- > >> ranlib -D libusb.a > >> ranlib: fatal: Failed to open 'libusb.a' > >> *** [libusb.a] Error code 70 > >> > >> and: > >> > >> --- all_subdir_lib/libbsnmp --- > >> ranlib: fatal: Failed to open 'libbsnmp.a' > >> --- all_subdir_lib/ncurses --- > >> --- all_subdir_lib/ncurses/panelw --- > >> --- panel.pico --- > >> --- all_subdir_lib/libbsnmp --- > >> *** [libbsnmp.a] Error code 70 > >> > >> > >> Even amd64 gets such: > >> > >> --- libpcap.a --- > >> ranlib -D libpcap.a > >> ranlib: fatal: Failed to open 'libpcap.a' > >> *** [libpcap.a] Error code 70 > >> > >> and: > >> > >> > >> --- libkafs5.a --- > >> ranlib: fatal: Failed to open 'libkafs5.a' > >> --- libkafs5_p.a --- > >> ranlib: fatal: Failed to open 'libkafs5_p.a' > >> --- cddl/lib__L --- > >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua/lbaselib.c:60:26: note: include the header or explicitly provide a declaration for 'toupper' > >> --- kerberos5/lib__L --- > >> *** [libkafs5_p.a] Error code 70 > >> > >> make[5]: stopped in /usr/src/kerberos5/lib/libkafs5 > >> --- libkafs5.a --- > >> *** [libkafs5.a] Error code 70 > >> > >> and: > >> > >> > >> --- lib__L --- > >> ranlib -D libclang_rt.asan_cxx-i386.a > >> ranlib: fatal: Failed to open 'libclang_rt.asan_cxx-i386.a' > >> *** [libclang_rt.asan_cxx-i386.a] Error code 70 > >> > >> > >> (Notice the variability in what .a the ranlib's fail for.) > >> > >> > >> > >> > >> > > > > > > I looked at this a few days ago and don't believe it's actually a build > > race. I think there is something wrong with the ar/ranlib on that system > > or something else. I've found no evidence of concurrent building of the > > .a files in question. > > > Looking at a bunch of the failures, spanning multiple > FreeBSD-head-*-build types of builds, I see only: > > NODE_LABELS bhyve_host butler1.nyi.freebsd.org jailer jailer_fast > NODE_NAME butler1.nyi.freebsd.org > > for the failures that I looked at. > > So your "on that system" might well be correct. Thanks for the insight, the build is done in a 11.1-R jail on a -CURRENT host. butler1.nyi is running r333388 (as a canary) while other builders are mostly running r328278. I upgraded few others and it seems can reproduce the issue, and now I downgraded all the build slaves to r328278 before we find the root cause. Li-Wen -- Li-Wen Hsu https://lwhsu.org