From nobody Wed Apr 24 04:30:41 2024 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VPQyt2nx1z5JkjC; Wed, 24 Apr 2024 04:30:46 +0000 (UTC) (envelope-from philip@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VPQyt1kdBz4Pct; Wed, 24 Apr 2024 04:30:46 +0000 (UTC) (envelope-from philip@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1713933046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jl8lMIN+aSxpWOmEo+9MvQAC+650q4O3M1+hwPe2JGk=; b=NzpWMljv8EkQ6/XpG7glRgZ1V5sVNa/PZybbm9bpBnFgxhSFcBQ/8rAp+yW+JDLcEU+eGo Bz99H/G2tdNU5jxBytiAn5jBkTdnD/OmgX+h156Vk88BO9ZtBIQuaVf7Xez6XfSItfo8Us /Tzm8BAZmTc5bdf7JZMOUzklz1OvJ+YcUByNeQiuQV/l9x3lRoTiQkCei+dH7r/44SrNSu n5z4sUSe/TDw4JtHkgUWyELah0kfv6x4yCObIRKVYUaMIE2t2d7Dr4RVqXtykVTpyKw1FC Gd1g2RB/cqP6JuPG2xVFw78X97CKetnbH+ExpfuukhJECsXyfj4sfdjSVFaV7Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1713933046; a=rsa-sha256; cv=none; b=sNI6GWzejqZvE3n71N+F5Q4d3KrVzQzVF1/gNKqa5bUrMVxYVzHfsxLoeH29cnzR2L9VA4 MEYmzd1OgMRL2l2oPU3o/awdQs9mtIVBAa7/pQbYHNHm5/t0kJ+kETbuBTahGgBGS585P9 jq4WBwY1a5kEP35YWA4tpmA/vBQX9NADLrsRylgAHTjEunwpxDqB1FxSVSJyMsebPvecKz 0JPWIzkn5xWa7eWjKhML8TzIaugju5kAFCKOAsKvQlf/y3UZVLQBDI4KHsVV5pLQHtSj+T tjDC5FBY+nhII0Vv5lPRi63YH7BakAtNmr9Tz9kgxuwerMeBTUPFf42GvrrfoA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1713933046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jl8lMIN+aSxpWOmEo+9MvQAC+650q4O3M1+hwPe2JGk=; b=XqfFyoz566N59sbM4V5wkAqjDrM2uYSaLWAvedtcY4lim1Yd7dUEeLwSrwCy4/NaIT2r0g S6htHn/QrnIqODNtIj41YARV77cure4O+wxF7o4Z8IcVV5CRT+8fSDFdZFBLIIpAvQ2+Fq cMEG8m+GtyJK656s8K5+Wfi5TJgImKrtccCbBnuRub8myx3gQ8QXJFQe73dilWNiYuLEIK 0CszhRV8ZCri0XgpklfUADWWJOCbkboeNSh3EzKmyQRwHVDut6NCe1O77XOrY5lxCSUOSA fAxJKdON2g0xssxDrgEX+jD2+NCvtWa8Sk28eM54baLBqnLYPujGy1sk8Cz9xw== Received: from fauth1-smtp.messagingengine.com (fauth1-smtp.messagingengine.com [103.168.172.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: philip/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4VPQyt12Scz18G6; Wed, 24 Apr 2024 04:30:46 +0000 (UTC) (envelope-from philip@freebsd.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfauth.nyi.internal (Postfix) with ESMTP id F2B401200043; Wed, 24 Apr 2024 00:30:45 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 24 Apr 2024 00:30:45 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrudelvddgjeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffokfgjfhggtgfgsehtqhhmtdertddtnecuhfhrohhmpefrhhhi lhhiphcurfgrvghpshcuoehphhhilhhiphesfhhrvggvsghsugdrohhrgheqnecuggftrf grthhtvghrnhepjedukeffgfffkeejfffgfefgledthefhffeggeevgeevhedvheegtddu vdetkeeinecuffhomhgrihhnpehfrhgvvggsshgurdhorhhgnecuvehluhhsthgvrhfuih iivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhhihhlihhpodhmvghsmhhtphgr uhhthhhpvghrshhonhgrlhhithihqdduudeiiedviedvgeekqddvfeehudektddtkedqph hhihhlihhppeepfhhrvggvsghsugdrohhrghesthhrohhusghlvgdrihhs X-ME-Proxy: Feedback-ID: ia691475d:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 24 Apr 2024 00:30:43 -0400 (EDT) From: Philip Paeps To: Mark Millard Cc: void , FreeBSD Mailing List , Current FreeBSD Subject: Re: pkg server for current/arm64 stopped ? [main-armv7 on ampere2, elapsed so far: 651:21:56] Date: Wed, 24 Apr 2024 12:30:41 +0800 X-Mailer: MailMate (1.14r6028) Message-ID: <6958AD93-6EDD-4D08-B446-63A9999D200A@freebsd.org> In-Reply-To: References: <03736C90-EE54-47B3-AEA7-ED1AC0343B4B.ref@yahoo.com> <03736C90-EE54-47B3-AEA7-ED1AC0343B4B@yahoo.com> List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-ports@freebsd.org Sender: owner-freebsd-ports@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; format=flowed Content-Transfer-Encoding: quoted-printable On 2024-04-24 02:12:41 (+0800), Mark Millard wrote: > On Apr 19, 2024, at 07:16, Philip Paeps wrote: > >> On 2024-04-18 23:02:30 (+0800), Mark Millard wrote: >> >>> void wrote on >>> Date: Thu, 18 Apr 2024 14:08:36 UTC : >>> >>>> Not sure where to post this.. >>>> >>>> The last bulk build for arm64 appears to have happened around >>>> mid-March on ampere2. Is it broken? >>> >>> main-armv7 building is broken and the last completed build >>> was the one started on Mon, 19 Feb 2024 12:32:10 GMT. It >>> gets stuck making no progress until manually forced to stop, >>> which leads to huge elapsed times for the incomplete builds: >>> >>> pd5512ae7b8c6_s75464941dc 34472 12282 (+9196) 107 (+77) 4753 = >>> (+2247) 1390 (+529) 15940 parallel_build: Fri, 22 Mar 2024 11:05:01 = >>> GMT 651:21:56 >>> >>> p43e3af5f5763_sf5f08e41aa 19809 5919 (+3126) 137 (+100) 5363 = >>> (+2741) 1395 (+522) 6995 parallel_build: Wed, 28 Feb 2024 15:46:14 = >>> GMT 359:42:14 ampere2 >>> >>> ampere2 alternates between trying to build main-arm64 and = >>> main-armv7, so main-armv7 being stuck blocks main-arm64 from = >>> building. >>> >>> One can see that all 13 job ID's show over 570 hours: >>> >>> http://ampere2.nyi.freebsd.org/build.html?mastername=3Dmain-armv7-def= ault&build=3Dpd5512ae7b8c6_s75464941dc >>> >>> It is not random which packages are building when this happens. = >>> Compare: >>> >>> http://ampere2.nyi.freebsd.org/build.html?mastername=3Dmain-armv7-def= ault&build=3Dp43e3af5f5763_sf5f08e41aa >>> >>> By contrast, the 19 Feb 2024 from-scratch (full) build worked: >>> >>> http://ampere2.nyi.freebsd.org/build.html?mastername=3Dmain-armv7-def= ault&build=3Dpe9c9c73181b5_sbd45bbe440 >>> >>> My guess is that FreeBSD has something that broken after bd45bbe440 >>> that was broken as of f5f08e41aa and was still broken at 75464941dc = >>> . >> >> I'll kill the build on ampere2 again. Thanks for the nudge. >> >> We don't really have good monitoring for this. Also: builds should = >> time out after 36 hours. The fact that this one does not is a bug in = >> itself. >> >> Philip [hat: clusteradm] > > I'll note that I've never managed to replicate the problem for > building for armv7 on aarch64. But my context never has the > likes of: > > QUOTE > Host OSVERSION: 1500006 > Jail OSVERSION: 1500015 > . . . > !!! Jail is newer than host. (Jail: 1500015, Host: 1500006) !!! > !!! This is not supported. !!! > !!! Host kernel must be same or newer than jail. !!! > !!! Expect build failures. !!! > END QUOTE > > but always has the two OSVERSION's the same, such as: > > Host OSVERSION: 1500015 > Jail OSVERSION: 1500015 > > or, recently, > > Host OSVERSION: 1500018 > Jail OSVERSION: 1500018 > > My bulk runs do go through the sequence where the hangups > have repeated for main-armv7 on ampere2. > > I wonder what would happen if "Host OSVERSION" was updated > (modernized) to match the modern "Jail OSVERSION" that would > be used? The package builders are due for a regular refresh to newer -CURRENT = dogfood. I'll do the aarch64 builders first this time. I've set /root/stop-builds on them. I'll upgrade them when they go = idle. Or I'll kill them if they take much longer to build what they're = building. It annoys me that they do not stop building after 36 hours, = like they're supposed to. They're currently running: n266879-6abee52e0d79 2023-12-09 01:06:28 jlduran strfmon: Silence = scan-build warning Our current clusteradm build is: n269399-bbc6e6c5ec8c 2024-04-14 03:12:36 sigsys daemon: fix -R to = enable supervision mode I may do a new build while waiting for them to go idle: - quarterly 140arm64 1b931669de11 parallel_build 28776 15299 33 588 = 985 0 11871 3D:01:08:29 = https://pkg-status.freebsd.org/ampere1/build.html?mastername=3D140arm64-q= uarterly&build=3D1b931669de11 - default main-arm64 p1c7a816cd0ad_s1bd4f769caf parallel_build 34528 = 19888 65 669 980 0 12926 4D:00:52:21 = https://pkg-status.freebsd.org/ampere2/build.html?mastername=3Dmain-arm64= -default&build=3Dp1c7a816cd0ad_s1bd4f769caf - default 140releng-armv7 2910ff97e727 parallel_build 34543 14826 60 = 5539 1397 0 12721 1D:09:35:28 = https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D140releng-= armv7-default&build=3D2910ff97e727 Philip