Date: Sat, 6 Dec 2025 15:25:36 -0700 From: Warner Losh <imp@bsdimp.com> To: Mark Millard <marklmi@yahoo.com> Cc: FreeBSD Current <freebsd-current@freebsd.org>, FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: performance regressions in 15.0 Message-ID: <CANCZdfqfXfzGQRN5TR7KFcNE1-Ng4ECFKD_6V0118b2UwwX09Q@mail.gmail.com> In-Reply-To: <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com> References: <EF95C136-B1D2-4820-A069-D0078A3B5A05@yahoo.com> <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Sat, Dec 6, 2025, 3:06 PM Mark Millard <marklmi@yahoo.com> wrote: > > On Dec 6, 2025, at 06:14, Mark Millard <marklmi@yahoo.com> wrote: > > > Mateusz Guzik <mjguzik_at_gmail.com> wrote on > > Date: Sat, 06 Dec 2025 10:50:08 UTC : > > > >> I got pointed at phoronix: > https://www.phoronix.com/review/freebsd-15-amd-epyc > >> > >> While I don't treat their results as gospel, a FreeBSD vs FreeBSD test > >> showing a slowdown most definitely warrants a closer look. > >> > >> They observed slowdowns when using iperf over localhost and when > compiling llvm. > >> > >> I can confirm both problems and more. > >> > >> I found the profiling tooling for userspace to be broken again so I > >> did not investigate much and I'm not going to dig into it further. > >> > >> Test box is AMD EPYC 9454 48-Core Processor, with the 2 systems > >> running as 8 core vms under kvm. > >> . . . > > > > > > > > Both of the below are from ampere3 (aarch64) instead, its > > 2 most recent "bulk -a" runs that completed, elapsed times > > shown for qt6-webengine-6.9.3 builds: > > > > 150releng-arm64-quarterly qt6-webengine-6.9.3 53:33:46 > > 135arm64-default qt6-webengine-6.9.3 38:43:36 > > > > For reference: > > > > Host OSVERSION: 1600000 > > Jail OSVERSION: 1500068 > > > > vs. > > > > Host OSVERSION: 1600000 > > Jail OSVERSION: 1305000 > > > > The difference for the above is in the Jail's world builds, > > not in the boot's (kernel+world) builds. > > > > > > For reference: > > > > > > > https://pkg-status.freebsd.org/ampere3/build.html?mastername=150releng-arm64-quarterly&build=88084f9163ae > > > > build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sun Nov 30 > 05:40:02 -00 2025 > > build time: 2D:05:33:52 > > > > > > > https://pkg-status.freebsd.org/ampere3/build.html?mastername=135arm64-default&build=f5384fe59be6 > > > > build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sat Nov 22 > 15:33:34 -00 2025 > > build time: 1D:14:43:41 > > > Expanding the notes to before and after jemalloc 5.3.0 > was merged to main: beefy18 was the main-amd64 builder > before and somewhat after the jemalloc 5.3.0 merge from > vendor branch: > > Before: p2650762431ca_s51affb7e971 261:29:13 building 36074 port-packages, > start 05 Aug 2025 01:10:59 GMT > ( jemalloc 5.3.0 merge from vendor > branch: 15 Aug 2025) > After : p9652f95ce8e4_sb45a181a74c 428:49:20 building 36318 port-packages, > start 19 Aug 2025 01:30:33 GMT > > (The log files are long gone for port-packages built.) > > main-15 used a debug jail world but 15.0-RELEASE does not. > > I'm not aware of such a port-package builder context for a > non-debug jail world before and after a jemalloc 5.3.0 merge. > A few months before I landed the jemalloc patches, i did 4 or 5 from dirt buildworlds. The elasped time was, iirc, with 1 or 2%. Enough to see maybe a diff with the small sample size, but not enough for ministat to trigger at 95%. I didn't recall keeping the data for this and can't find it now. And I'm not even sure, in hindsight, I ran a good experiment. It might be related, or not, but it would be easy enough for someone to setup a two jails: one just before and one just after. Build from scratch the world (same hash) on both. That would test it since you'd be holding all other variables constant. When we imported the tip of FreeBSD main at work, we didn't get a cpu change trigger from our tests that I recall... Warner Warner > [-- Attachment #2 --] <div dir="auto"><div><br><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Sat, Dec 6, 2025, 3:06 PM Mark Millard <<a href="mailto:marklmi@yahoo.com">marklmi@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br> On Dec 6, 2025, at 06:14, Mark Millard <<a href="mailto:marklmi@yahoo.com" target="_blank" rel="noreferrer">marklmi@yahoo.com</a>> wrote:<br> <br> > Mateusz Guzik <<a href="http://mjguzik_at_gmail.com" rel="noreferrer noreferrer" target="_blank">mjguzik_at_gmail.com</a>> wrote on<br> > Date: Sat, 06 Dec 2025 10:50:08 UTC :<br> > <br> >> I got pointed at phoronix: <a href="https://www.phoronix.com/review/freebsd-15-amd-epyc" rel="noreferrer noreferrer" target="_blank">https://www.phoronix.com/review/freebsd-15-amd-epyc</a><br>; >> <br> >> While I don't treat their results as gospel, a FreeBSD vs FreeBSD test<br> >> showing a slowdown most definitely warrants a closer look.<br> >> <br> >> They observed slowdowns when using iperf over localhost and when compiling llvm.<br> >> <br> >> I can confirm both problems and more.<br> >> <br> >> I found the profiling tooling for userspace to be broken again so I<br> >> did not investigate much and I'm not going to dig into it further.<br> >> <br> >> Test box is AMD EPYC 9454 48-Core Processor, with the 2 systems<br> >> running as 8 core vms under kvm.<br> >> . . .<br> > <br> > <br> > <br> > Both of the below are from ampere3 (aarch64) instead, its<br> > 2 most recent "bulk -a" runs that completed, elapsed times<br> > shown for qt6-webengine-6.9.3 builds:<br> > <br> > 150releng-arm64-quarterly qt6-webengine-6.9.3 53:33:46<br> > 135arm64-default qt6-webengine-6.9.3 38:43:36<br> > <br> > For reference:<br> > <br> > Host OSVERSION: 1600000<br> > Jail OSVERSION: 1500068<br> > <br> > vs.<br> > <br> > Host OSVERSION: 1600000<br> > Jail OSVERSION: 1305000<br> > <br> > The difference for the above is in the Jail's world builds,<br> > not in the boot's (kernel+world) builds.<br> > <br> > <br> > For reference:<br> > <br> > <br> > <a href="https://pkg-status.freebsd.org/ampere3/build.html?mastername=150releng-arm64-quarterly&build=88084f9163ae" rel="noreferrer noreferrer" target="_blank">https://pkg-status.freebsd.org/ampere3/build.html?mastername=150releng-arm64-quarterly&build=88084f9163ae</a><br>; > <br> > build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sun Nov 30 05:40:02 -00 2025<br> > build time: 2D:05:33:52<br> > <br> > <br> > <a href="https://pkg-status.freebsd.org/ampere3/build.html?mastername=135arm64-default&build=f5384fe59be6" rel="noreferrer noreferrer" target="_blank">https://pkg-status.freebsd.org/ampere3/build.html?mastername=135arm64-default&build=f5384fe59be6</a><br>; > <br> > build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sat Nov 22 15:33:34 -00 2025<br> > build time: 1D:14:43:41<br> <br> <br> Expanding the notes to before and after jemalloc 5.3.0<br> was merged to main: beefy18 was the main-amd64 builder<br> before and somewhat after the jemalloc 5.3.0 merge from<br> vendor branch:<br> <br> Before: p2650762431ca_s51affb7e971 261:29:13 building 36074 port-packages, start 05 Aug 2025 01:10:59 GMT<br> ( jemalloc 5.3.0 merge from vendor branch: 15 Aug 2025)<br> After : p9652f95ce8e4_sb45a181a74c 428:49:20 building 36318 port-packages, start 19 Aug 2025 01:30:33 GMT<br> <br> (The log files are long gone for port-packages built.)<br> <br> main-15 used a debug jail world but 15.0-RELEASE does not.<br> <br> I'm not aware of such a port-package builder context for a<br> non-debug jail world before and after a jemalloc 5.3.0 merge.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">A few months before I landed the jemalloc patches, i did 4 or 5 from dirt buildworlds. The elasped time was, iirc, with 1 or 2%. Enough to see maybe a diff with the small sample size, but not enough for ministat to trigger at 95%. I didn't recall keeping the data for this and can't find it now. And I'm not even sure, in hindsight, I ran a good experiment. It might be related, or not, but it would be easy enough for someone to setup a two jails: one just before and one just after. Build from scratch the world (same hash) on both. That would test it since you'd be holding all other variables constant.</div><div dir="auto"><br></div><div dir="auto">When we imported the tip of FreeBSD main at work, we didn't get a cpu change trigger from our tests that I recall...</div><div dir="auto"><br></div><div dir="auto">Warner</div><div dir="auto"><br></div><div dir="auto">Warner</div><div dir="auto"><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> </blockquote></div></div></div>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfqfXfzGQRN5TR7KFcNE1-Ng4ECFKD_6V0118b2UwwX09Q>