Date: Sat, 19 Aug 2023 03:20:23 +0000 From: Glen Barber <gjb@freebsd.org> To: Mark Millard <marklmi@yahoo.com> Cc: Current FreeBSD <freebsd-current@freebsd.org> Subject: Re: Possible regression in main causing poor performance Message-ID: <20230819032023.GB4090@FreeBSD.org> In-Reply-To: <8A267B4A-1B7B-4D61-AA57-8E3156470617@yahoo.com> References: <8A267B4A-1B7B-4D61-AA57-8E3156470617.ref@yahoo.com> <8A267B4A-1B7B-4D61-AA57-8E3156470617@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--ZoaI/ZTpAVc4A5k6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 18, 2023 at 07:09:10PM -0700, Mark Millard wrote: > Glen Barber <gjb_at_freebsd.org> wrote on > Date: Sat, 19 Aug 2023 00:10:59 UTC : >=20 > > I am somewhat inclined to look in the direction of ZFS here, as two > > things changed: > >=20 > > 1) the build machine in question was recently (as in a week and a half > > ago) upgraded to the tip of main in order to ease the transition from > > this machine from building 14.x to building 15.x; > > 2) there is the recent addition of building ZFS-backed virtual machine > > and cloud images. > >=20 > > . . . > > The first machine runs: > > # uname -a > > FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \ > > amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023 >=20 > I'm confused: >=20 > "the build machine in question was recently (as in a week and a half > ago) upgraded to the tip of main in order to ease the transition from > this machine from building 14.x to building 15.x"? But the above > kernel is from mid July? (-aKU was not used to also get some clue > about world from the pair of 140009? that would show.) >=20 Err, right. I'm confused too, and apparently looked at a different machine but copied 'uname -a' from the correct machine. The kernel and userland are always in sync on these machines. # uname -UK 1400093 1400093 > > Last week's snapshot builds were completed in a reasonable amount of > > time: > >=20 > > root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./= builds-14.conf ; echo ^G > > 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs > > 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots > > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release > > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports > > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc > > 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git (= main) to /releng/scripts-snapshot/release > > [...] > > 20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap > > 20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap > > 20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap > >=20 > > Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE. > >=20 > > With no changes to the system, no stale ZFS datasets laying around from > > last week (everything is a pristine environment, etc.), this week's > > builds are taking forever: >=20 > My confusion may extend to this "no changes" status vs. the uname > output identifying the kernel is from mid July. >=20 It is admittedly a month versus two weeks difference here. My fault. > > root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./= builds-14.conf ; echo ^G > > 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs > > 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots > > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release > > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports > > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc > > 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git (= main) to /releng/scripts-snapshot/release > > [...] > > 20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap > > 20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap > > 20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap > >=20 > > Note, it is just about 4 minutes past 00:00 UTC as of this writing, so > > we are about to cross well over the 24-hour mark, and cloud provider > > images have not yet even started. > >=20 > > . . . >=20 > In: >=20 > https://lists.freebsd.org/archives/freebsd-current/2023-August/004314.html > ("HEADS UP: $FreeBSD$ Removed from main", Wed, 16 Aug 2023) >=20 > Warner wrote: >=20 > QUOTE > . . . , but there's no incremental building > with this change, . . . Also: expect long build times, git fetch times, e= tc > after this. > END QUOTE >=20 > Might this be contributing? How long did those two > "Checking out . . ." take? Similar time frames? >=20 Checkout times are negligible. They are pristine environments, so there are no delta resolutions, etc. Everything is clean from the start. Last week: 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.g= it (main) to /releng/scripts-snapshot/release 20230811-00:05:27 INFO: Creating ZFS snapshot zroot/releng/14-src-s= nap@clone 20230811-00:05:33 INFO: Checking out https://git.FreeBSD.org//ports= =2Egit (main) to /releng/scripts-snapshot/ports 20230811-00:09:44 INFO: Creating ZFS snapshot zroot/releng/14-ports= -snap@clone This week: 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.g= it (main) to /releng/scripts-snapshot/release 20230818-00:17:57 INFO: Creating ZFS snapshot zroot/releng/14-src-s= nap@clone 20230818-00:18:01 INFO: Checking out https://git.FreeBSD.org//ports= =2Egit (main) to /releng/scripts-snapshot/ports 20230818-00:22:09 INFO: Creating ZFS snapshot zroot/releng/14-ports= -snap@clone Glen --ZoaI/ZTpAVc4A5k6 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmTgNPIACgkQAxRYpUeP 4pN53Q//bN6NBJwQkG/nuGkJMq6xxnkTpfAb+GMmBewQDZv8Hc+PI+cCZJP6I0FN aQ0D5LDRoeoYjkB5+ZIsX8eaEO+LpEm9eJoUhBJBx/e1xRE7AOA5KDMIo5Ce6Mvq 3oICTZSBQR9CRO6EqJt4oN0x9zx6vOPbNE6neHDIpuaFBVqiFhyzV/S0JfmnRJ0V QNzaCgvEwiN+8UyAY+6ayAqCfBmE6WEs2+ZmbWKRQrwX4n5WsLYgll3pYliEErgS TQRotn5F4fRo9ogNh8qtj3mZsT2jg27zzetvBRQC0zhRiZ6CKSkBxhNnAJ6V5/SW L+P8/N2wyLsL3EeQiJjVcUSkvKrrFmTQ0nQ8eYLD5ppLq5P4z+aXbNy5dmzec+rt p1LcKuX/hLjYjpkP+mtyxEW11z0FPFfxiMt7rI2Z0EzewjiVWFn+pvefQ40rHCB1 7pLWaTMwRrwCMXPtYx3CvQ1uPsTxZAVVdp5RL3qU6Y406EX8m9xxiOKcTDMB3d75 guzJ2d7rbP2VetkDsEgO81zMQ18/TtLDmShMFoGAjSgxLStOM0PCW5J0ZXBxNnZQ bYAyEVD7NMOug/RjfbngrQjflAxDhwlMosj8n47jDNCLdeWaIXZ4q2BQYL7YE43n 7FLxVpziMuh3brdbKm1Aq+3l8EdzFg02elxuoowkvZp3V87ePAk= =pAkC -----END PGP SIGNATURE----- --ZoaI/ZTpAVc4A5k6--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20230819032023.GB4090>