From owner-freebsd-toolchain@freebsd.org Mon Jun 18 22:31:29 2018 Return-Path: Delivered-To: freebsd-toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CEA310020AC; Mon, 18 Jun 2018 22:31:29 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B37BA77249; Mon, 18 Jun 2018 22:31:28 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-wm0-f65.google.com with SMTP id x6-v6so16598201wmc.3; Mon, 18 Jun 2018 15:31:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8RpHs30NV5Os7ugVYxDPl5HhR5Nvs7RTdw3sWSVNngY=; b=ePrPh2J7n5K/Bf9zzsC/XhRTSW5yaeLLY8qT+33U2ePw8b/f+Tp/u7j5AQga9XEhLz 2KLSQxd619oiNxSSQZk2zTVg0w2olJBbG5kDcNY/3Orz5pPJMi7i5FbduYo17jn51d82 roH04PkdJC1I6liPgKrbpwCmAGuy8JAYhALTVuz/TaekZu0nbSeqIu4NSxJG2mjr9vJz U5bIAJeqzjF+9eC4t5Lv+rMQabfjwsp4WEqU4r8eRyMW1U9OL5OMCR2VeyvV026utuIe 4J3x1yw2l8o/nHPmT8dZA8e3NW8KW3vBylbQfv7mmfTuWF9KuRz5Xllw/10LjpM3HtVP 4ddw== X-Gm-Message-State: APt69E1eIF2C4TnUyDu1U2E94OQXld5JCObongcrnq54ydpvWhLnmbp9 D8OdoQ2X0g3YsfsGRAbKNBU0BKDXbREvNqeMYYOD9qw4 X-Google-Smtp-Source: ADUXVKLrtUIuSNTtr9+7R7HOfHKI+HxF2ie6yruw4BPO+jHgWLwxiCmu0bUcYZITbJm3rcllUlby5x0JNfXn573WVR4= X-Received: by 2002:a1c:42d7:: with SMTP id k84-v6mr9549420wmi.159.1529361087390; Mon, 18 Jun 2018 15:31:27 -0700 (PDT) MIME-Version: 1.0 References: <74EAD684-0E0B-453A-B746-156777CF604A@yahoo.com> <1884103f-d1fb-aca6-2edd-062e11d05617@FreeBSD.org> <20180618204517.GD2430@kib.kiev.ua> <068108ab-76f2-0f2d-fd92-11c838a4d391@FreeBSD.org> In-Reply-To: <068108ab-76f2-0f2d-fd92-11c838a4d391@FreeBSD.org> From: Li-Wen Hsu Date: Mon, 18 Jun 2018 18:31:14 -0400 Message-ID: Subject: Re: A head buildworld race visible in the ci.freebsd.org build history To: Bryan Drewery Cc: Konstantin Belousov , FreeBSD Current , FreeBSD Toolchain Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 22:31:29 -0000 On Mon, Jun 18, 2018 at 6:27 PM Bryan Drewery wrote: > > On 6/18/2018 1:45 PM, Konstantin Belousov wrote: > > On Mon, Jun 18, 2018 at 12:42:46PM -0700, Bryan Drewery wrote: > >> On 6/15/2018 10:55 PM, Mark Millard wrote: > >>> In watching ci.freebsd.org builds I've seen a notable > >>> number of one time failures, such as (example from > >>> powerpc64): > >>> > >>> --- all_subdir_lib/libufs --- > >>> ranlib -D libufs.a > >>> ranlib: fatal: Failed to open 'libufs.a' > >>> *** [libufs.a] Error code 70 > >>> > >>> where the next build works despite the change being > >>> irrelevant to whatever ranlib complained about. > >>> > >>> Other builds failed similarly: > >>> > >>> --- all_subdir_lib/libbsm --- > >>> ranlib -D libbsm_p.a > >>> ranlib: fatal: Failed to open 'libbsm_p.a' > >>> *** [libbsm_p.a] Error code 70 > >>> > >>> and: > >>> > >>> --- kerberos5/lib__L --- > >>> ranlib -D libgssapi_spnego_p.a > >>> --- libgssapi_spnego.a --- > >>> ranlib -D libgssapi_spnego.a > >>> --- libgssapi_spnego_p.a --- > >>> ranlib: fatal: Failed to open 'libgssapi_spnego_p.a' > >>> *** [libgssapi_spnego_p.a] Error code 70 > >>> > >>> and so on. > >>> > >>> > >>> It is not limited to powerpc64. For example, for aarch64 > >>> there are: > >>> > >>> --- libpam_exec.a --- > >>> building static pam_exec library > >>> ar -crD libpam_exec.a `NM='nm' NMFLAGS='' lorder pam_exec.o | tsort -q` > >>> ranlib -D libpam_exec.a > >>> ranlib: fatal: Failed to open 'libpam_exec.a' > >>> *** [libpam_exec.a] Error code 70 > >>> > >>> and: > >>> > >>> --- all_subdir_lib/libusb --- > >>> ranlib -D libusb.a > >>> ranlib: fatal: Failed to open 'libusb.a' > >>> *** [libusb.a] Error code 70 > >>> > >>> and: > >>> > >>> --- all_subdir_lib/libbsnmp --- > >>> ranlib: fatal: Failed to open 'libbsnmp.a' > >>> --- all_subdir_lib/ncurses --- > >>> --- all_subdir_lib/ncurses/panelw --- > >>> --- panel.pico --- > >>> --- all_subdir_lib/libbsnmp --- > >>> *** [libbsnmp.a] Error code 70 > >>> > >>> > >>> Even amd64 gets such: > >>> > >>> --- libpcap.a --- > >>> ranlib -D libpcap.a > >>> ranlib: fatal: Failed to open 'libpcap.a' > >>> *** [libpcap.a] Error code 70 > >>> > >>> and: > >>> > >>> > >>> --- libkafs5.a --- > >>> ranlib: fatal: Failed to open 'libkafs5.a' > >>> --- libkafs5_p.a --- > >>> ranlib: fatal: Failed to open 'libkafs5_p.a' > >>> --- cddl/lib__L --- > >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua/lbaselib.c:60:26: note: include the header or explicitly provide a declaration for 'toupper' > >>> --- kerberos5/lib__L --- > >>> *** [libkafs5_p.a] Error code 70 > >>> > >>> make[5]: stopped in /usr/src/kerberos5/lib/libkafs5 > >>> --- libkafs5.a --- > >>> *** [libkafs5.a] Error code 70 > >>> > >>> and: > >>> > >>> > >>> --- lib__L --- > >>> ranlib -D libclang_rt.asan_cxx-i386.a > >>> ranlib: fatal: Failed to open 'libclang_rt.asan_cxx-i386.a' > >>> *** [libclang_rt.asan_cxx-i386.a] Error code 70 > >>> > >>> > >>> (Notice the variability in what .a the ranlib's fail for.) > >>> > >>> > >>> > >>> > >>> > >> > >> > >> I looked at this a few days ago and don't believe it's actually a build > >> race. I think there is something wrong with the ar/ranlib on that system > >> or something else. I've found no evidence of concurrent building of the > >> .a files in question. > > > > FWIW, I got the similar failure when I did last checks for the OFED > > commit. For me, it was libgcc.a. > > > > If it was -lgcc_s then it's a known rare build race due to > tools/install.sh not handling -S. It seems a more general problem, this one: https://ci.freebsd.org/job/FreeBSD-head-aarch64-build/8190/console calls for libcuse_p.a, while this one: https://ci.freebsd.org/job/FreeBSD-head-mips-build/2919/console calls for libfifolog.a -- Li-Wen Hsu https://lwhsu.org