From nobody Sat Dec 6 22:25:36 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dP2rZ3p2Qz6JlxV for ; Sat, 06 Dec 2025 22:25:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dP2rZ0X5Kz49CY for ; Sat, 06 Dec 2025 22:25:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-ba599137cf8so1242785a12.0 for ; Sat, 06 Dec 2025 14:25:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1765059948; x=1765664748; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=2JkMm96sdgudqDpCZhA2O9LaDKUt0Tfv6kZZR+uDHJc=; b=f4qB5YGTANt61VnDmfuff2nyK00ayA0JPQ/CBj7/MfKtAkMhgYNs3rnTO+g7s9oCyq r9T3vZBY5/4kFVX5MEptrAVDqBYgClWmNFNhUYemOCOjHJ5BIu48wTjxtO6uqb7AHjf6 clx1gfzV5il0ccVgDLY/rj+no2blT+4apjV8JCSRa54v5Mig7kAuSP47/ymWMho5fBrE JyUgoE/e0LNq2OBiEySkB4XYDwBGHlv/xGf2zY9KgY1ZT7AkEm2y20+kb+EvCu0rTUZI Jo/4jpYLm6vokadV4kALNB5tVezVr4WOTD+4QJ8htdjNhj/PmXJbo7H3bAMTHcU3lB+t M4QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765059948; x=1765664748; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2JkMm96sdgudqDpCZhA2O9LaDKUt0Tfv6kZZR+uDHJc=; b=dsiP0qmJO27BnamaaeSmPetlNzuNXNH7eSSMEZ8KrLnbwxWmkqGWhiyLgnKMMoJ8hs OfBldT1cNbVEqvXMbF/b2TXbl73R1gmKxpPa4RSwvw8rot5WsxjPfpPkXeM12005Jsup xiMNWG5eNNTPj2VEW2AzNsfzybG5dxwElH0wml75UltuK77leKm1vq32Tmty4Snx1Bd9 97HYyofoieD4fDrA2vlVqCS4AFaCM/eDTms7GHeeuyCvp2/pJ8xq+30/cJZul2nGoX1t MkOCJcbJUrbLZ7JKF4G5D2V5Pz25qIMaZEeiLkTmpk9aeI5AiIL3sAm+wIFZMZRl6ivO 08PA== X-Gm-Message-State: AOJu0YypId4uou9H2edNmyLdFS6i7nmGQsPNl07mCWByk9tO+RAq3Vfm X3os+KuU+j63KedjTxqmRSMK36Wz49aK2734p+NZG/f6eF6w3KgSkmILnEqzU+ob09SlIUEJwhD jhtkKhSU1VGbA1HazfV84qQMY+ha21tLd6N+BbmuhFQ== X-Gm-Gg: ASbGncshql1r/8dunjKQygWCVQFoG42diQHXzStWuadABGlh5HNi1WFCnA/4COAhiQT YKegyN6izD7bCxB1tUDdYk/U+WxJWvLjojxdFRnwZUMDACm4wl6vkNu0aQqHbg7y7GZjajVlcfq UPcKdL0ZOJy2mZT4hW2cQO0QjFWPlTAjbowSLuadqVFwD3gcqkXj4qmvVISZr/L1u69u9RJ400k 4aEE0sy43F9A73YSjpynOGPGb0x+/OtALUy5p1fgmwn0JXCWxGZLlrzYLCkWzkZ1eH1rEOjWCCw tJLJAJ1/NkU7qAQiqSc8klZFow== X-Google-Smtp-Source: AGHT+IEQKmV5K15+M2ZlUcLRk9AI0cfTOspraauJTR2Qg47s10ZtfYIo07mqqIvZ1Ktz4wNJa75cdB5ecqZpnIoPVGA= X-Received: by 2002:a17:90b:3a47:b0:343:7711:127d with SMTP id 98e67ed59e1d1-349438d603fmr10253103a91.9.1765059947571; Sat, 06 Dec 2025 14:25:47 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com> In-Reply-To: <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com> From: Warner Losh Date: Sat, 6 Dec 2025 15:25:36 -0700 X-Gm-Features: AQt7F2pUxFU9xqMSssQVSZE3drTpEfSqhDgzrBltmd5Q_VX1EyHCDymfq4etiHQ Message-ID: Subject: Re: performance regressions in 15.0 To: Mark Millard Cc: FreeBSD Current , FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="0000000000002ebbee0645500bc9" X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dP2rZ0X5Kz49CY --0000000000002ebbee0645500bc9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Dec 6, 2025, 3:06=E2=80=AFPM Mark Millard wrote= : > > On Dec 6, 2025, at 06:14, Mark Millard wrote: > > > Mateusz Guzik wrote on > > Date: Sat, 06 Dec 2025 10:50:08 UTC : > > > >> I got pointed at phoronix: > https://www.phoronix.com/review/freebsd-15-amd-epyc > >> > >> While I don't treat their results as gospel, a FreeBSD vs FreeBSD test > >> showing a slowdown most definitely warrants a closer look. > >> > >> They observed slowdowns when using iperf over localhost and when > compiling llvm. > >> > >> I can confirm both problems and more. > >> > >> I found the profiling tooling for userspace to be broken again so I > >> did not investigate much and I'm not going to dig into it further. > >> > >> Test box is AMD EPYC 9454 48-Core Processor, with the 2 systems > >> running as 8 core vms under kvm. > >> . . . > > > > > > > > Both of the below are from ampere3 (aarch64) instead, its > > 2 most recent "bulk -a" runs that completed, elapsed times > > shown for qt6-webengine-6.9.3 builds: > > > > 150releng-arm64-quarterly qt6-webengine-6.9.3 53:33:46 > > 135arm64-default qt6-webengine-6.9.3 38:43:36 > > > > For reference: > > > > Host OSVERSION: 1600000 > > Jail OSVERSION: 1500068 > > > > vs. > > > > Host OSVERSION: 1600000 > > Jail OSVERSION: 1305000 > > > > The difference for the above is in the Jail's world builds, > > not in the boot's (kernel+world) builds. > > > > > > For reference: > > > > > > > https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D150releng-= arm64-quarterly&build=3D88084f9163ae > > > > build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sun Nov 30 > 05:40:02 -00 2025 > > build time: 2D:05:33:52 > > > > > > > https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D135arm64-d= efault&build=3Df5384fe59be6 > > > > build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sat Nov 22 > 15:33:34 -00 2025 > > build time: 1D:14:43:41 > > > Expanding the notes to before and after jemalloc 5.3.0 > was merged to main: beefy18 was the main-amd64 builder > before and somewhat after the jemalloc 5.3.0 merge from > vendor branch: > > Before: p2650762431ca_s51affb7e971 261:29:13 building 36074 port-packages= , > start 05 Aug 2025 01:10:59 GMT > ( jemalloc 5.3.0 merge from vendor > branch: 15 Aug 2025) > After : p9652f95ce8e4_sb45a181a74c 428:49:20 building 36318 port-packages= , > start 19 Aug 2025 01:30:33 GMT > > (The log files are long gone for port-packages built.) > > main-15 used a debug jail world but 15.0-RELEASE does not. > > I'm not aware of such a port-package builder context for a > non-debug jail world before and after a jemalloc 5.3.0 merge. > A few months before I landed the jemalloc patches, i did 4 or 5 from dirt buildworlds. The elasped time was, iirc, with 1 or 2%. Enough to see maybe a diff with the small sample size, but not enough for ministat to trigger at 95%. I didn't recall keeping the data for this and can't find it now. And I'm not even sure, in hindsight, I ran a good experiment. It might be related, or not, but it would be easy enough for someone to setup a two jails: one just before and one just after. Build from scratch the world (same hash) on both. That would test it since you'd be holding all other variables constant. When we imported the tip of FreeBSD main at work, we didn't get a cpu change trigger from our tests that I recall... Warner Warner > --0000000000002ebbee0645500bc9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Sat, Dec 6, 2025, 3:06=E2=80= =AFPM Mark Millard <marklmi@yahoo.c= om> wrote:

On Dec 6, 2025, at 06:14, Mark Millard <marklmi@yahoo.com> wrote:<= br>
> Mateusz Guzik <mjguzik_at_gmail.com> wrote on > Date: Sat, 06 Dec 2025 10:50:08 UTC :
>
>> I got pointed at phoronix: ht= tps://www.phoronix.com/review/freebsd-15-amd-epyc
>>
>> While I don't treat their results as gospel, a FreeBSD vs Free= BSD test
>> showing a slowdown most definitely warrants a closer look.
>>
>> They observed slowdowns when using iperf over localhost and when c= ompiling llvm.
>>
>> I can confirm both problems and more.
>>
>> I found the profiling tooling for userspace to be broken again so = I
>> did not investigate much and I'm not going to dig into it furt= her.
>>
>> Test box is AMD EPYC 9454 48-Core Processor, with the 2 systems >> running as 8 core vms under kvm.
>> . . .
>
>
>
> Both of the below are from ampere3 (aarch64) instead, its
> 2 most recent "bulk -a" runs that completed, elapsed times > shown for qt6-webengine-6.9.3 builds:
>
> 150releng-arm64-quarterly qt6-webengine-6.9.3 53:33:46
> 135arm64-default=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 qt6-webengine-6.9.3= 38:43:36
>
> For reference:
>
> Host OSVERSION: 1600000
> Jail OSVERSION: 1500068
>
> vs.
>
> Host OSVERSION: 1600000
> Jail OSVERSION: 1305000
>
> The difference for the above is in the Jail's world builds,
> not in the boot's (kernel+world) builds.
>
>
> For reference:
>
>
> https://pkg-status.freebsd.org/ampere3/build.h= tml?mastername=3D150releng-arm64-quarterly&build=3D88084f9163ae
>
> build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sun Nov 30 0= 5:40:02 -00 2025
> build time: 2D:05:33:52
>
>
> https://pkg-status.freebsd.org/ampere3/build.html?maste= rname=3D135arm64-default&build=3Df5384fe59be6
>
> build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sat Nov 22 1= 5:33:34 -00 2025
> build time: 1D:14:43:41


Expanding the notes to before and after jemalloc 5.3.0
was merged to main: beefy18 was the main-amd64 builder
before and somewhat after the jemalloc 5.3.0 merge from
vendor branch:

Before: p2650762431ca_s51affb7e971 261:29:13 building 36074 port-packages, = start 05 Aug 2025 01:10:59 GMT
(=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0jemalloc = 5.3.0 merge from vendor branch: 15 Aug 2025)
After : p9652f95ce8e4_sb45a181a74c 428:49:20 building 36318 port-packages, = start 19 Aug 2025 01:30:33 GMT

(The log files are long gone for port-packages built.)

main-15 used a debug jail world but 15.0-RELEASE does not.

I'm not aware of such a port-package builder context for a
non-debug jail world before and after a jemalloc 5.3.0 merge.

A few months b= efore I landed the jemalloc patches, i did 4 or 5 from dirt buildworlds. Th= e elasped time was, iirc, with 1 or 2%. Enough to see maybe a diff with the= small sample size, but not enough for ministat to trigger at 95%. I didn&#= 39;t recall keeping the data for this and can't find it now. And I'= m not even sure, in hindsight, I ran a good experiment. It might be related= , or not, but it would be easy enough for someone to setup a two jails: one= just before and one just after. Build from scratch the world (same hash) o= n both. That would test it since you'd be holding all other variables c= onstant.

When we importe= d the tip of FreeBSD main at work, we didn't get a cpu change trigger f= rom our tests that I recall...

Warner

Warner
--0000000000002ebbee0645500bc9--