From nobody Sat Aug 19 03:20:23 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RSPBh61Bbz4q49T for ; Sat, 19 Aug 2023 03:20:28 +0000 (UTC) (envelope-from gjb@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RSPBh5Mt5z3Xys; Sat, 19 Aug 2023 03:20:28 +0000 (UTC) (envelope-from gjb@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692415228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kqHhSiq6pqUxB/RvMCpT/+9kOpPv9RK1rM/lSBBhjIM=; b=hUdtHyUzPpbm6HhsarCMbWMCL1sYcjkTgBoVRSiXonugJaC9x0KzfZFQziF+NA8Xm3z37P CnoxiUZrtaTWEaePn2UcU5ECZXPmcJ8db8VnqZNhICWydWPB9qAIWCe3fs3xlyj1ypFzic oNev1ML+IPtgpI7gERJvsBWurM6EpwFtKywvG7Guejx8w2vh133EjM85FdWVX+Lj/WxUP+ RSToBt61zT1GoLW+KgB1/O+eP/9mttW/hUBcMxhGx42WQMnNmw4PmHD2bkO6/rIsCLNFjp UP0RBuC61mEzIphybZ/KCQjbHLY2Z3X77PjuSSpjDXCJv1jllnyz9bQAoBNHgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692415228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kqHhSiq6pqUxB/RvMCpT/+9kOpPv9RK1rM/lSBBhjIM=; b=cIL7vwLQ0BjbXvF/1BLvlvLXPHw4Re7AWUlL+O8izSQQeTOQayAXLz6L37CCbDC6v5knSg S/LsArtTUnL8e4ennr1hMH0YTv9pPmr0RIWl3wXGmQ0E4AlfpV/xWO/OI4awSH7++4cepU z5tlH+SKtY79/P0HYmsWwz451Ov0FEIq8lhvKG/szKfLwFujl4r2jUamBMbg9fJ/CVno2F OPswN0ybplDhgwX2OBNswP1HKShTro0HhrloEySuF/LAerKtU50D/iJZYWS1hwAlbnegl2 T5JVg0uhPwu7xIoLoQQDECbtZk3r5Jv4r4Ycuo2+IcJpwbX5OSjbPL2+08zghg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692415228; a=rsa-sha256; cv=none; b=r9fs+Z8aQZzhzfuN4VTyNGGZfOyCH5feYocFpopnlMXE3yRu7Pa8zQrgn2ht8bDBaBg1QZ 0N/ecxcCHfxGzXD4satJUEieZhnlf6iS61b/Jn+z7FAqBkoEuIvKDcri7WyoI2av41B102 Lm3IJ3kIhcAN1h37cAD6NbWyAT8eX6soib1shzaZWn9YwB31XPoqFMm2LeD1Dc7spkmUk6 9EYAWrdKTem43zRJD8sg2NlJJyD/ZIZQyTCSsuG7WQo5LL1/PzWCTY6hu+Kh3qyEKzcylj RBnawOUCQkvvci2U5SpmTbB8kGofNTPiZYYvm6ViHiiS+KjM5m4VEy0QXT3L9A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 727C812B00; Sat, 19 Aug 2023 03:20:28 +0000 (UTC) (envelope-from gjb@freebsd.org) Date: Sat, 19 Aug 2023 03:20:23 +0000 From: Glen Barber To: Mark Millard Cc: Current FreeBSD Subject: Re: Possible regression in main causing poor performance Message-ID: <20230819032023.GB4090@FreeBSD.org> References: <8A267B4A-1B7B-4D61-AA57-8E3156470617.ref@yahoo.com> <8A267B4A-1B7B-4D61-AA57-8E3156470617@yahoo.com> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ZoaI/ZTpAVc4A5k6" Content-Disposition: inline In-Reply-To: <8A267B4A-1B7B-4D61-AA57-8E3156470617@yahoo.com> --ZoaI/ZTpAVc4A5k6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 18, 2023 at 07:09:10PM -0700, Mark Millard wrote: > Glen Barber wrote on > Date: Sat, 19 Aug 2023 00:10:59 UTC : >=20 > > I am somewhat inclined to look in the direction of ZFS here, as two > > things changed: > >=20 > > 1) the build machine in question was recently (as in a week and a half > > ago) upgraded to the tip of main in order to ease the transition from > > this machine from building 14.x to building 15.x; > > 2) there is the recent addition of building ZFS-backed virtual machine > > and cloud images. > >=20 > > . . . > > The first machine runs: > > # uname -a > > FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \ > > amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023 >=20 > I'm confused: >=20 > "the build machine in question was recently (as in a week and a half > ago) upgraded to the tip of main in order to ease the transition from > this machine from building 14.x to building 15.x"? But the above > kernel is from mid July? (-aKU was not used to also get some clue > about world from the pair of 140009? that would show.) >=20 Err, right. I'm confused too, and apparently looked at a different machine but copied 'uname -a' from the correct machine. The kernel and userland are always in sync on these machines. # uname -UK 1400093 1400093 > > Last week's snapshot builds were completed in a reasonable amount of > > time: > >=20 > > root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./= builds-14.conf ; echo ^G > > 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs > > 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots > > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release > > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports > > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc > > 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git (= main) to /releng/scripts-snapshot/release > > [...] > > 20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap > > 20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap > > 20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap > >=20 > > Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE. > >=20 > > With no changes to the system, no stale ZFS datasets laying around from > > last week (everything is a pristine environment, etc.), this week's > > builds are taking forever: >=20 > My confusion may extend to this "no changes" status vs. the uname > output identifying the kernel is from mid July. >=20 It is admittedly a month versus two weeks difference here. My fault. > > root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./= builds-14.conf ; echo ^G > > 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs > > 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots > > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release > > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports > > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc > > 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git (= main) to /releng/scripts-snapshot/release > > [...] > > 20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap > > 20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap > > 20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap > >=20 > > Note, it is just about 4 minutes past 00:00 UTC as of this writing, so > > we are about to cross well over the 24-hour mark, and cloud provider > > images have not yet even started. > >=20 > > . . . >=20 > In: >=20 > https://lists.freebsd.org/archives/freebsd-current/2023-August/004314.html > ("HEADS UP: $FreeBSD$ Removed from main", Wed, 16 Aug 2023) >=20 > Warner wrote: >=20 > QUOTE > . . . , but there's no incremental building > with this change, . . . Also: expect long build times, git fetch times, e= tc > after this. > END QUOTE >=20 > Might this be contributing? How long did those two > "Checking out . . ." take? Similar time frames? >=20 Checkout times are negligible. They are pristine environments, so there are no delta resolutions, etc. Everything is clean from the start. Last week: 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.g= it (main) to /releng/scripts-snapshot/release 20230811-00:05:27 INFO: Creating ZFS snapshot zroot/releng/14-src-s= nap@clone 20230811-00:05:33 INFO: Checking out https://git.FreeBSD.org//ports= =2Egit (main) to /releng/scripts-snapshot/ports 20230811-00:09:44 INFO: Creating ZFS snapshot zroot/releng/14-ports= -snap@clone This week: 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.g= it (main) to /releng/scripts-snapshot/release 20230818-00:17:57 INFO: Creating ZFS snapshot zroot/releng/14-src-s= nap@clone 20230818-00:18:01 INFO: Checking out https://git.FreeBSD.org//ports= =2Egit (main) to /releng/scripts-snapshot/ports 20230818-00:22:09 INFO: Creating ZFS snapshot zroot/releng/14-ports= -snap@clone Glen --ZoaI/ZTpAVc4A5k6 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmTgNPIACgkQAxRYpUeP 4pN53Q//bN6NBJwQkG/nuGkJMq6xxnkTpfAb+GMmBewQDZv8Hc+PI+cCZJP6I0FN aQ0D5LDRoeoYjkB5+ZIsX8eaEO+LpEm9eJoUhBJBx/e1xRE7AOA5KDMIo5Ce6Mvq 3oICTZSBQR9CRO6EqJt4oN0x9zx6vOPbNE6neHDIpuaFBVqiFhyzV/S0JfmnRJ0V QNzaCgvEwiN+8UyAY+6ayAqCfBmE6WEs2+ZmbWKRQrwX4n5WsLYgll3pYliEErgS TQRotn5F4fRo9ogNh8qtj3mZsT2jg27zzetvBRQC0zhRiZ6CKSkBxhNnAJ6V5/SW L+P8/N2wyLsL3EeQiJjVcUSkvKrrFmTQ0nQ8eYLD5ppLq5P4z+aXbNy5dmzec+rt p1LcKuX/hLjYjpkP+mtyxEW11z0FPFfxiMt7rI2Z0EzewjiVWFn+pvefQ40rHCB1 7pLWaTMwRrwCMXPtYx3CvQ1uPsTxZAVVdp5RL3qU6Y406EX8m9xxiOKcTDMB3d75 guzJ2d7rbP2VetkDsEgO81zMQ18/TtLDmShMFoGAjSgxLStOM0PCW5J0ZXBxNnZQ bYAyEVD7NMOug/RjfbngrQjflAxDhwlMosj8n47jDNCLdeWaIXZ4q2BQYL7YE43n 7FLxVpziMuh3brdbKm1Aq+3l8EdzFg02elxuoowkvZp3V87ePAk= =pAkC -----END PGP SIGNATURE----- --ZoaI/ZTpAVc4A5k6--