Date: Sat, 19 Aug 2023 00:10:59 +0000 From: Glen Barber <gjb@freebsd.org> To: freebsd-current@freebsd.org, freebsd-arch@freebsd.org, freebsd-stable@freebsd.org Cc: FreeBSD Release Engineering Team <re@freebsd.org> Subject: Possible regression in main causing poor performance Message-ID: <20230819001059.GH1219@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
--f/XZMZU9ST6S8MjH
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
I am somewhat inclined to look in the direction of ZFS here, as two
things changed:
1) the build machine in question was recently (as in a week and a half
ago) upgraded to the tip of main in order to ease the transition from
this machine from building 14.x to building 15.x;
2) there is the recent addition of building ZFS-backed virtual machine
and cloud images.
Here is a bit of the history of the build machines:
- the second and first build 13.x and 14.x, respectively. The third is
(at the time it was purchased) newer than the other two, and
significantly fast when it comes to weekly snapshots. I'll refer to
these machines as such: first, second, and third.
- The third machine will be eventually used to build 14.0-RELEASE. The
will first continue on main, where it will build 15.0-CURRENT
snapshots. The second will continue tracking 13-STABLE.
The first machine runs:
# uname -a
FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \
amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023
Last week's snapshot builds were completed in a reasonable amount of
time:
root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./builds-14.conf ; echo ^G
20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs
20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots
20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release
20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports
20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc
20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release
[...]
20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap
20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap
20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap
Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE.
With no changes to the system, no stale ZFS datasets laying around from
last week (everything is a pristine environment, etc.), this week's
builds are taking forever:
root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./builds-14.conf ; echo ^G
20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs
20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots
20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release
20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports
20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc
20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release
[...]
20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap
20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap
20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap
Note, it is just about 4 minutes past 00:00 UTC as of this writing, so
we are about to cross well over the 24-hour mark, and cloud provider
images have not yet even started.
I am inclined to do two things:
1) Immediately run a subsequent snapshot build to see if it takes longer
than 24 hours (my gut tells me it will);
2) Reboot the machine with no other changes, and immediately run this
week's snapshot builds again (not to be public to avoid confusion).
Some interactive commands are significantly slower, such as systat,
vmstat, even top. Creating a new window in tmux is also noticeably
slow.
Is there a third option I am overlooking in trying to identify the
drastic cause of this?
Thank you in advance.
Glen
--f/XZMZU9ST6S8MjH
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmTgCI4ACgkQAxRYpUeP
4pOKtg/8CQioy5hX+3kT5uGpdv1IokbSJ+Wg7vhaB99RuG92RbtZNufmgh1bzbuj
GyBD9vkwfNExjhOYRnvPRoDpwZC4cufxIANKi0Cwl6DKxTCrMOF3UGM3hM2dY4lm
7bniYolDgzm0v0GzAhgafi0q5Znsm4o4dslm0iUdLqj6ISH2fFI3FJoMQ+moDt/c
vCNo8wz0BgCLA3BVILHzwdjoecksONLyoa9K8xYE/YfI2uXt/RrRzcHdw5qGKyKA
eKpV6MBZF3ITawIt8rPUWp0/WGPun+KBW/uD0QumDW1VCMH+tLpyuB6vh4EtCOyj
KOnTp1zqUBoLN3dxnwi58wDOy+WA++QAHcRB80UmuCYgU1pGYAO0aox77c88prb2
VGpAf81TnLJDiB+Ibphyxx0ju8pujQtTbywu1wmFDur4dtmfrv6ZZTTWwkb6lucD
TygH4y2fQqzgD9Qcx/2nOH7Ax0KPN7/W4xi+O0537PcNAzJP2UM+gaXnzmn1/HTn
aRkmLy3oOTlL3mkSi9R12t0B6TpmVbRL8cUrBMtA4et/n4v+7eMXsKgVn4JWYqv+
vB02mHrZFSDsb6jV0z+T/F8LHm5NMmAGohfpHCXZCtJ6VI5Y/350Faslj5bznVyA
5DyCXizNyxJgCAvagCRX3G6Rg/B5HytlHUnK4s9qYSDMHRNin80=
=WC7c
-----END PGP SIGNATURE-----
--f/XZMZU9ST6S8MjH--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20230819001059.GH1219>