From nobody Sat Aug 19 00:10:59 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RSK0B4yNYz4fw8g; Sat, 19 Aug 2023 00:11:06 +0000 (UTC) (envelope-from gjb@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RSK0B4Tpjz4svm; Sat, 19 Aug 2023 00:11:06 +0000 (UTC) (envelope-from gjb@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692403866; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=PhHInTX0oaR8Loj6VGOfRH0eHjN/iSY5nx1cjy6ANNg=; b=RCgExbtPqeuz1tpbhoo4AzNjdt78pVX0bx0UEyKBPXVqVw2vVeVlky5KSCgaLS0B7Njqf9 MwotHErg6GFjgsQF3k98Kk+NSH+J87hefzM6Uhk31PR//IRQzJRMHTxTBKQm4As0ZUf/6v 8iwU3xBUC6Pb/yr11M2oPqobw2DrsUzJJKp192OfSjwuNgCDm4P1efD5DxbKX/syZpdIEa GXtIQjK8xeiqT+1UHxMrFa+WZNsJz1wQn9qToMGW4QsJo3mV0dluN8XpW33HhRThCFaB7R Ab56itu1Y52lMQxh66dsG0J46xymOt5v1RFyRuBv/kM2649r36cUVH14Vp62PQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692403866; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=PhHInTX0oaR8Loj6VGOfRH0eHjN/iSY5nx1cjy6ANNg=; b=CHNYRmQjVWU4ra6pO+nxIjhKfri+wKXjXe3+9Osy+YmD3EA9kl8A2Edd27uBm5qibnUopN woLee3XpN1P2YxyHBvJKMnL26HqN854ceTvhfPBMas91Xaol41oeOkg09WsFs4YJrqjcIU oqM01r9lG/ZYNAgdEyh12WpxLrwQ4M9+3xK+KqzaYKnqIwBmMveTeP79UYzaLtu7KsPjXZ 4AJI3FCJ2wEJDIZJuYGa2g3O03iVGyjR6CrxpQp4UVKvVQoF++LyaJk/PzwNkg+e21ZD/0 lqEF0owOXdaal39TRmqqfaeY6+vIIgD0e24wM3vtDnq++Fv5loTUUfLJeHWjIQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692403866; a=rsa-sha256; cv=none; b=pg+qJkczX/53WmhJ6qUKYzfgqjzJNk/Fa2jjsbjmj8zRbgOustaS4JqwodynDXCRdeLpri cOaEDLc7yPxZeeTHomYZREV/VROuZS3lIhjaWm0HLdSvtCLwaLBoZpcASW+s+MFhGc8uCd O9f9uK87uDpVbO7yMCuhnrS1wJoUBQytEE0WrFevLEoRX4L9Hk+FonChe+Ng3P4Cd9tb0c sHGIyIvszRtCWr8NAOaD9OLa3B+zB0eopHtgz7hLxr4dFWyYxOgM8YVxh3hR55B3eQLVWC qJs+ky/8BAb1Frk0UheoP6gsMzdK+S6GlU8nJNui5gX+izNO/zr+s0pdgdQ4Hg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id CF3ED12185; Sat, 19 Aug 2023 00:11:05 +0000 (UTC) (envelope-from gjb@freebsd.org) Date: Sat, 19 Aug 2023 00:10:59 +0000 From: Glen Barber To: freebsd-current@freebsd.org, freebsd-arch@freebsd.org, freebsd-stable@freebsd.org Cc: FreeBSD Release Engineering Team Subject: Possible regression in main causing poor performance Message-ID: <20230819001059.GH1219@FreeBSD.org> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="f/XZMZU9ST6S8MjH" Content-Disposition: inline --f/XZMZU9ST6S8MjH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline I am somewhat inclined to look in the direction of ZFS here, as two things changed: 1) the build machine in question was recently (as in a week and a half ago) upgraded to the tip of main in order to ease the transition from this machine from building 14.x to building 15.x; 2) there is the recent addition of building ZFS-backed virtual machine and cloud images. Here is a bit of the history of the build machines: - the second and first build 13.x and 14.x, respectively. The third is (at the time it was purchased) newer than the other two, and significantly fast when it comes to weekly snapshots. I'll refer to these machines as such: first, second, and third. - The third machine will be eventually used to build 14.0-RELEASE. The will first continue on main, where it will build 15.0-CURRENT snapshots. The second will continue tracking 13-STABLE. The first machine runs: # uname -a FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \ amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023 Last week's snapshot builds were completed in a reasonable amount of time: root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./builds-14.conf ; echo ^G 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release [...] 20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap 20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap 20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE. With no changes to the system, no stale ZFS datasets laying around from last week (everything is a pristine environment, etc.), this week's builds are taking forever: root@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c ./builds-14.conf ; echo ^G 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git (main) to /releng/scripts-snapshot/release [...] 20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap 20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap 20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap Note, it is just about 4 minutes past 00:00 UTC as of this writing, so we are about to cross well over the 24-hour mark, and cloud provider images have not yet even started. I am inclined to do two things: 1) Immediately run a subsequent snapshot build to see if it takes longer than 24 hours (my gut tells me it will); 2) Reboot the machine with no other changes, and immediately run this week's snapshot builds again (not to be public to avoid confusion). Some interactive commands are significantly slower, such as systat, vmstat, even top. Creating a new window in tmux is also noticeably slow. Is there a third option I am overlooking in trying to identify the drastic cause of this? Thank you in advance. Glen --f/XZMZU9ST6S8MjH Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAmTgCI4ACgkQAxRYpUeP 4pOKtg/8CQioy5hX+3kT5uGpdv1IokbSJ+Wg7vhaB99RuG92RbtZNufmgh1bzbuj GyBD9vkwfNExjhOYRnvPRoDpwZC4cufxIANKi0Cwl6DKxTCrMOF3UGM3hM2dY4lm 7bniYolDgzm0v0GzAhgafi0q5Znsm4o4dslm0iUdLqj6ISH2fFI3FJoMQ+moDt/c vCNo8wz0BgCLA3BVILHzwdjoecksONLyoa9K8xYE/YfI2uXt/RrRzcHdw5qGKyKA eKpV6MBZF3ITawIt8rPUWp0/WGPun+KBW/uD0QumDW1VCMH+tLpyuB6vh4EtCOyj KOnTp1zqUBoLN3dxnwi58wDOy+WA++QAHcRB80UmuCYgU1pGYAO0aox77c88prb2 VGpAf81TnLJDiB+Ibphyxx0ju8pujQtTbywu1wmFDur4dtmfrv6ZZTTWwkb6lucD TygH4y2fQqzgD9Qcx/2nOH7Ax0KPN7/W4xi+O0537PcNAzJP2UM+gaXnzmn1/HTn aRkmLy3oOTlL3mkSi9R12t0B6TpmVbRL8cUrBMtA4et/n4v+7eMXsKgVn4JWYqv+ vB02mHrZFSDsb6jV0z+T/F8LHm5NMmAGohfpHCXZCtJ6VI5Y/350Faslj5bznVyA 5DyCXizNyxJgCAvagCRX3G6Rg/B5HytlHUnK4s9qYSDMHRNin80= =WC7c -----END PGP SIGNATURE----- --f/XZMZU9ST6S8MjH--