From nobody Sat Mar 26 23:15:55 2022 X-Original-To: toolchain@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7D1591A4054E for ; Sat, 26 Mar 2022 23:16:11 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-20.consmr.mail.gq1.yahoo.com (sonic313-20.consmr.mail.gq1.yahoo.com [98.137.65.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KQvwB3MtZz4bHn for ; Sat, 26 Mar 2022 23:16:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1648336563; bh=VYO3sDSwPlD6eXKloTlfxhmXOIyXFAPWSoKp5HuPU3s=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=YYGJww+r46vPkJSEs7J2xAAwH/HVG5MgRLdLrTcQrnUwlPjCLZi+9nFQitLxqkdcCgTYNldZiLEcYKfxZ15+bFS0h1vDO+KS4xPoEEfqpgUOhauTJKz8YA3sGT/gTDuXmIXNNCXkT/MOO/eDR3nXFJLUlsnmLDPblikwrya84bGaWyBQA65AOcKkGGt5dvIsw0ZwjWtZ4//+f/kwlmZtKIFIgmP8fTqVHVjCfAVlKNBj0bqVQClo4tLfH3XwsXBOBbq1NKegU4VXYoeetlYHVtCAGthrCDbhUME+OqaOctIf8buREgqrLtQFK2S2ET73/cXGkrERz7SZl859vxS7yg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1648336563; bh=zVrcsqY/Wh1aYdQiiX4IEDEnonkPCSQFzIQCfxijlKV=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=WjEWHgQwqp5PPy7TGGvviPRoc7H1hKcin6zPGjLj0UNgxhv1ukBhBh4Nkp6RtPzoJvsSGg55eASy6mLkepPqHUgSn1Ym4UI8vvUYi5qqas2nh/izz7ztZWBSf8XkodoBVi7Qckyb0z3MGZw5ihCtLb6CMwXoWZ+th3EE5LAarKhcO8ycES7CJbdaVufbU8XRlSnZ1VDs61950IZEwc9ptZ8G0GiMePthppwihbYJRgjTW0jalVy5qjVnNbs4xx0f+KagqQueIZK2EDIYLNeDR9C/LumFfO11sosmiy0zyj6tVDzBt261B9La8lGiuKejvIWRcPkBMmM0RiH1wKQbZg== X-YMail-OSG: i7M4vp8VM1nf2J4hfJB1yqZ.yZci2miAuP4cIzTdLx7D9cHnWeZ49mvRALZ9qf6 tKarxWNWUv5vzN6Yfq3qhcTch1_3jxe1hszew4ZqAQVn5HLCvw9k4_ojnQ6qNVjzfsmjGSeKhMiM B2NYE7xWAxk5u4s3TmXtExuV36dLvBL1bWLoj6yMe949oDl5I1rDeJK6JfBVPVKnb87QvmOxN5bK 7l743IvjO592BLUqhCrO9FYg1in0C1eIIG5fMVSWDE2fXsISa_yXU8Hv99PLTGp63QFshVnQN2Qr .bFA.DS7hvisRg.5c0FiIKlBi8Fn3HIph1lsf.t5j48C06ieJ3pcjU3NYH2Sv2PCi6J3Wqwa3czG O1Z1K6vrObOTayDzyU7HBy6DTrjfQ0QQQae3zBdiHrb2Eh1O74sroYFRBXC6PTmwn693yCAg2t08 SFJG9lvjLaooWNdl2nLL0AVZucIB0OC8k1rAGYAR3uGy2dQhkYoAnewnP1WQE7CX5eYtZePowi5r R.SSACdZRr5beYQA_Hd2EER6j9_OMDNVeDjde.uIuW0k.OI_hKmoouvSEt9Ynsv8_xWKMB.fy8hF Q8UF4JgvtoZ4XjgxcfqclHwjYgI9TcwOhApmzAcmQSfMqG4jo5Y5fTx4bB8Rk.CEldPVkiS3te5q sv6bx4qoSjbz0PS5b2Pdtw5G5XorLRACacMLnHE9.FLbtlR6rZJluS9p9OnmrRhsYEXmFXsgInnL oMlBPQgM7IYy1rlOmz5m096bV8CZz.fd8JIR_2SFwnr41i14eZyCU7QrInMDnqYC_8l0XueJPL.i 8LgMPaV_YYEy4bcGincfHzQzAu6CPRQaAWLP.E9CZWGVB4bGRUsxXoLlrRcH1qMGAxspN.pFePVm i6zYycamAATbPPtRlWb2ya8D.B_79_UbklfKHZIxL621KgV9R1rfNCwGwmoi2HBWpF5UkIseA9ld gngKeuhWk1Q6aQZoPn7ovOTDyK4l1HSacqS_z8fUzfyF0rye.uu04rv_DF8LOwnfhCFyX80C1xsf lqOHA3ig4cX9lzLn8np15418b_rPaS1NSewx.frcHHfpgP4nX4mpWpjUDoCuEa3uRr3P.DGFucJn balbuHy5XlbO.YjqhJFIX_BdI28BtL2fE8bjv_TJ3MtzsvjccdIOeR8jDwrt7fTSjPrq_cd7tdP_ n3kzdu1QAmIkbmRKlm8qYvGfh6zIr2emDKGyXlMssdpVA.OavW9nlioSWwcdfzwvdyWWsUftNob8 QdNu2nrJRoQt.shziHHn77U53iFXsfJtm0UzOg660PxNon1l4CQNgv.NjtYISbr5mpLAmHocDlL0 d..PORn1C7kV2D3dpQpfA0Oul_kc42qDirWPEznrMJ9adIYD4fj1sOz80Ons3TEDQ4aCsjLCgG9b Vri4iixMFJnRVZWuNE1_rwkHQi_bTjKdkn2nciHjtRnYK1s_I80r5N11ReoBBjGHkbWdemQiZYoL xZCBwDYvVIqwULGgiIoYOcaxuEEDR2q9TUJKaAIM_EtnieKsOotbSWuEBNKJu2JTrTn5aClZaQfA JJZGhrl5VkRtJLD1q6V_FucvzYSpBb_GgEv1DHwJvz.6metqdY6X_XHXdUis_hdpqxHWIJLWD.WG Eef5LVgR.FbHxDJ6pLrZppyYu6OA3Z3sdibt0IBSkERFsVTvOkTCQp3IptH58w7fkeCITO3wfNz6 33jAyBwfu3GE947dKpkhi1hnb.A3Bm.7fs3VDKTmyGpJVxFKU7kZZQxOEcpFOWFvbsDpTB9iFQPl ACRlPRAkpzzJ7huHIJHOkzCW_Xl5wDG7jrzkBxbaUgvJQL0eY8L8zUdtSmZ45FlSb8gvzr6uNNUG zZ_GJPncgscvd8kkU_l8TQOsVDK7dfk2gIW1WsdNPhNBlMXJT07CRPyk1._bybjk8ANdNxmgV45h _.qGyBE8dhqKnuxExvBE8rnoMgJvfmjQbzon8SBlXn8ADgzadV7FePiRK1D6Pwh4gIev0KUqTx26 olWwqdeogS4R1pbHVm1_ZfLyLs5V0SK6eioeZCk0yoigxpYwcP7.DYE.gSe03hud6.zcVQRfjgMu cdFnxPq0dQeDNK5_YQbHSMyOzs_IrUyoVgw2QAK_4bVshzwgltFNVsNXn2r0RdefT4QBstwUtwBg tNI0rMyxf3NokU76mfotWcqHSqFYJYp7Y7lMq_ehUG01aAD0T0a3HlZdyScNn14TlMCJ8MNGO.al OID1itQvJ6PAdg_.fqD4JPZZuI02vmAQ7fG_TVQxfrLHigEBaWYTW5cYFcnKyz5u3i81ryntRUrm dWnfdpFCZQDxYlwpFSxAka97zEyazgU5lB2G7VmSdMomMnCAMj9BXHs3e56uYCW5JA8XAL53jdNB 8p463tgoupvkY70L8JYLUeNgP4V27iQeeoiFed0bWBjBWMQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Sat, 26 Mar 2022 23:16:03 +0000 Received: by hermes--canary-production-bf1-665cdb9985-zm65g (VZM Hermes SMTP Server) with ESMTPA ID 52b42f68410dd4ac4a3632b467f28b93; Sat, 26 Mar 2022 23:15:57 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Maintenance of FreeBSD s integrated toolchain List-Archive: https://lists.freebsd.org/archives/freebsd-toolchain List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-toolchain@freebsd.org X-BeenThere: freebsd-toolchain@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: [package - 130arm64-default][lang/gcc12-devel] Failed for gcc12-devel-12.0.1.s20220306_2 in build/runaway From: Mark Millard In-Reply-To: Date: Sat, 26 Mar 2022 16:15:55 -0700 Cc: "toolchain@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <17CAD266-C7C0-4CD7-B255-3DC07F422EB5@yahoo.com> References: <202203261416.22QEGtRR065106@ampere3.nyi.freebsd.org> <21D1C2BF-151E-4252-936C-B5B22C9C8071@yahoo.com> <75A61EB5-70D1-4E1F-89D2-524407854D6F@yahoo.com> To: Dimitry Andric X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4KQvwB3MtZz4bHn X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=YYGJww+r; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-2.98 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.48)[-0.483]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.83:from]; MLMMJ_DEST(0.00)[toolchain]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.83:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Mar-26, at 14:37, Mark Millard wrote: > On 2022-Mar-26, at 13:16, Mark Millard wrote: >=20 >> On 2022-Mar-26, at 12:35, Mark Millard wrote: >>=20 >>> On 2022-Mar-26, at 07:26, Dimitry Andric wrote: >>>=20 >>>> On 26 Mar 2022, at 15:16, pkg-fallout@freebsd.org = wrote: >>>>>=20 >>>>> You are receiving this mail as a port that you maintain >>>>> is failing to build on the FreeBSD package build server. >>>>> Please investigate the failure and submit a PR to fix >>>>> build. >>>>>=20 >>>>> Maintainer: toolchain@FreeBSD.org >>>>> Log URL: = http://ampere3.nyi.freebsd.org/data/130arm64-default/60ab72786154/logs/gcc= 12-devel-12.0.1.s20220306_2.log >>>>> Build URL: = http://ampere3.nyi.freebsd.org/build.html?mastername=3D130arm64-default&bu= ild=3D60ab72786154 >>>>=20 >>>> So there isn't any actual error message in this log, except at the = end: >>>>=20 >>>> ... >>>> =3D>> Cleaning up wrkdir >>>> =3D=3D=3D> Cleaning for gcc12-devel-12.0.1.s20220306_2 >>>> Killed >>>> build of lang/gcc12-devel | gcc12-devel-12.0.1.s20220306_2 ended at = Sat Mar 26 14:16:58 UTC 2022 >>>> build time: 12:31:35 >>>> !!! build failure encountered !!! >>>>=20 >>>> It looks like the last command being run before "Killed" is the = cc1plus >>>> executable being linked with LTO, so I am assuming the build is = killed >>>> due to an out-of-memory condition? >>>>=20 >>>> But this is only visible to people that have access to the machine = the >>>> poudriere instance is running on. Can somebody with access please = check? >>>>=20 >>>=20 >>> I do not have access but I've started a poudriere build >>> of my own on a HoneyComb. I've a patched top that monitors >>> and reports various Maximum Observed (MaxObs????) figures, >>> 64 GiBytes of RAM and slightly over 246 GiBytes of swap. >>> So hopefully it will report on about how big the memory use >>> gets. But it is allowed to use all 16 cores and there will >>> be no competing bulk builds using resources. So not a match >>> to the build server context. >>>=20 >>> Note: It is a ZFS context, so MaxObsWired is normally large >>> and shrinks over times where memory needs to be used for >>> other things. So the primary memory figures would be: >>>=20 >>> MaxObsSwapUsed (if any) >>> MaxObsActive >>> MaxObs(Act+Lndry+SwapUsed) >>>=20 >>>=20 >>> Side Note: >>>=20 >>> = http://ampere3.nyi.freebsd.org/build.html?mastername=3D130arm64-default&bu= ild=3D60ab72786154 >>>=20 >>> reports a Time of 11:48:41 but the log reports "build time: = 12:31:35". >>> My guess is that processing the log file for extracting the type of >>> error makes some (much?) of the difference. (Type being = runaway_process >>> in this case.) >>>=20 >>>=20 >>=20 >> I did just observe a cc1plus take somewhat over 30min >> of CPU time before completing and the lto1 related activity >> starting. It was under 5 GiBytes MaxObs(Act+Lndry+SwapUsed) >> [No swap use observed] before the lto1 related activity >> started. >>=20 >> For the lto1 related activity MaxObs(Act+Lndry+SwapUsed) >> has, so far, gotten up to around 12 GiBytes, still >> no swap use observed: >>=20 >> 12079Mi MaxObsActive >> 12278Mi MaxObs(Act+Lndry+SwapUsed) >>=20 >> I'll note that: >>=20 >> last pid: . . .; load averages: . . . MaxObs: 28.02, 16.88, = 15.82 >> . . . threads: . . . running, . . . sleeping, 77 MaxObsRunning >>=20 >> So, on the timescale of the first load average, it does >> not always stay limited to the hardware threads available. >>=20 >> No process with sustained CPU activity sticks around across >> the lto1 activity. So I'll not be able to observe much about >> cpu time. >>=20 >> The elasped time doing lto1 activity has been going for a >> while but I'm unlikely to be able to observe its end happen. >> So I'll likely not have a good clue about that. >>=20 >=20 > Looks it spent about 1.5 or so hours on the particular block of > lto1 related activity. For reference, somewhat after that: >=20 > last pid: . . .; load averages: . . . MaxObs: 28.02, 17.04, 16.87 >=20 > The 16 core are Cortex-A72's. >=20 > The following did not change (so far): >=20 > 12079Mi MaxObsActive > 12278Mi MaxObs(Act+Lndry+SwapUsed) >=20 > Still no observed swap use reported.=20 >=20 >=20 > The build is continuing. The build phase has been a little > over 2.5 hr so far. >=20 I'll note that: = http://ampere1.nyi.freebsd.org/build.html?mastername=3D130arm64-default&bu= ild=3Dd09582c49c48 reports for lang/gcc12-devel a build time of 21:53:01 . (So lang/gcc12-devel has built after LTO was enabled.) Back to my lang/gcc12-devel build . . . The earlier large block of lto1 activity was from 7 .usr/.local/bin/ld = 's going in parallel. (I had a ps -auxd output in a file to look at.) = Another block of 7 is going and taking significant time. (Seen via top.) [Waiting for it to get past that block of lto1 related activity. . . .] Hmm. I was not there to observe the end of the lto1 activity and what immediately followed. It is now in package: [00:26:46] [01] [00:00:00] Building lang/gcc12-devel | = gcc12-devel-12.0.1.s20220306_1 . . . load: 1.36 cmd: sh 10124 [nanslp] 15151.01r 21.72u 70.93s 0% 4096k mi_switch+0x104 sleepq_catch_signals+0x41c sleepq_timedwait_sig+0x14 = _sleep+0x1c0 kern_clock_nanosleep+0x1ac sys_nanosleep+0x3c = do_el0_sync+0x5cc handle_el0_sync+0x40=20 [13_0R-CA72-default] [2022-03-26_11h31m18s] [parallel_build:] Queued: 19 = Built: 18 Failed: 0 Skipped: 0 Ignored: 0 Fetched: 0 Tobuild: 1 = Time: 04:12:31 ID TOTAL ORIGIN PKGNAME PHASE = PHASE TMPFS CPU% MEM% [01] 03:45:51 lang/gcc12-devel | gcc12-devel-12.0.1.s20220306_1 package = 00:08:12 32.00 KiB 100% 0.2% [04:12:37] Logs: = /usr/local/poudriere/data/logs/bulk/13_0R-CA72-default/2022-03-26_11h31m18= s So somewhat over 3 hrs to reach the package stage. The following have not changed since the prior reported figures: last pid: . . .; load averages: . . . MaxObs: 28.02, 17.04, 16.87 . . . threads: . . . running, . . . sleeping, 77 MaxObsRunning and: 12079Mi MaxObsActive 12278Mi MaxObs(Act+Lndry+SwapUsed) (No report of observed swap usage.) [Waiting for the "Finished" message from poudriere. . . .] [04:33:13] [01] [04:06:27] Finished lang/gcc12-devel | = gcc12-devel-12.0.1.s20220306_1: Success =3D=3D=3D Mark Millard marklmi at yahoo.com