From nobody Mon Dec 8 12:46:45 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dQ1vm07Bvz6Jl7S for ; Mon, 08 Dec 2025 12:47:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dQ1vl5GS6z3Wkk for ; Mon, 08 Dec 2025 12:46:59 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52f.google.com with SMTP id 4fb4d7f45d1cf-640aa1445c3so6430294a12.1 for ; Mon, 08 Dec 2025 04:46:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1765198018; x=1765802818; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=lYHWZ2YyoSp49j882sGwkbDn5s7wYiwtkoSazaMeWaI=; b=NqPXA2s/HA1vjxP2n+XmzWHCw1yH34snvMt1Qewxz4vF6mypEqbfNm+D3x/W/c1iLa ol7PlFVjmOzLGTY6favVfJ2PfoEOyj38VlQJvCiTqHvGlvjhrwY8Kyo2dnLChZ82EvGa gGvS7efPQtVjfr2weBp51hd+3+3qw4J2qgriy6CzaS003iMVbKsdV7fIYASI/4lteUbo Nk3AJuJTh72Pkq1WmpajbI7acghCJgGB0PJtrCfPJn1+hIVba14VWhsSzm0skaboQ5mo qZHrJkMuffT22qt0ejPdIqXwSbBdtxQ7EOBJknnUUscj8YqNFsZxahtxxw9hvj4NIXKh SLDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765198018; x=1765802818; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=lYHWZ2YyoSp49j882sGwkbDn5s7wYiwtkoSazaMeWaI=; b=G3KjB8b1SiEFHMm9H2aToEALsq5XWWZOnMLOmuZ8U+fnXkZoaPb8jTwt+Nc55aFmLo GWfkVwBZ/x4LKW62LkeLdlPQzXNRg8bCC7UMieDJSUb4TrJoW5hXaNFUSvIAy9y5sJ91 B8pjlVgKkHYden4GJDy88NJPIvqfd8DOSvLtXUt69d1kNmqRi1Tr2mx4iiJgAF7Y4uRT Xe+5y5TvkviJi6+DMok69wQdyG0HWMqDeoOd3+TCIbm739H6Ot0C8pBbS3hrQnyDbyW7 Lt3XMhF48TOFsvNL6YU3ZVHvW7XhRV5VEERqQPYe4sYEQkMGLF6Uky9Z/KKmPk0nxPVT CXNA== X-Forwarded-Encrypted: i=1; AJvYcCW+4Did24jakMQmcEg63jzEu2osfQa8npX29eOnwEmDFBKFspZenT0qKK8iJVmEW4k+Q6+CREL2SO3ZVu5EMQg=@freebsd.org X-Gm-Message-State: AOJu0Yzpny6HslfXzrMSj+QKtMNmvJ98hHqvPflsnvPRpsfQWjzh3Uqz yVUrNS7Vyn3/rbHuriV2kluTIDKYEIdKH5JqkG01CaywEM849z/fqAOBwOGbwGTxJjY8+16onMC UbrK77ZKivp6t2zMrHmP2ODAiT1gBkhK4APOk X-Gm-Gg: ASbGncvCTjPQVNtrqg0YiW1Bt9jOmrBf7paYLpzvn14yOuEd44bV88sCH3YR4y4OyCc YOOXqT3R2B6hWEwYUIlSaUNiy+xH/7ciLD9I4TVLEaDZHE0utF9aQ5UIz667kOVWZqGr1V9+F2x qXWdvFTpiEIDdrTdR9rR/Q/4BxApMcNUrRMouz7cIYYlOeMEDe7oz3fFe1ElLMlucqpz8fITLRz NYJaTXrm9QpJtDoLBgDXln8Yy57gNst1wZNiIFgV+jdGW1/ihmzeAFL+Z8ZJCZrQJe9oATy5oTl U2gnYB7+n32MW9TnZ4Uhd/hI4A== X-Google-Smtp-Source: AGHT+IEgYql+2xUjMhNxEs3O/dM8oV7a/FY3y/l6HAdOenD3z8AVyymgwNssiicTAC+OMGw5Bx0PQZYS16sRip6ZCE0= X-Received: by 2002:a05:6402:1470:b0:649:2cc6:854 with SMTP id 4fb4d7f45d1cf-6492cc60a99mr4525129a12.10.1765198017714; Mon, 08 Dec 2025 04:46:57 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <18FB2858-5CBB-4B7A-8089-224A58C6A160@yahoo.com> <19A848A6-0042-4873-B70D-AD6805225B92@yahoo.com> <902C948B-0A4C-48E1-8C6C-1BC7A15209D7@yahoo.com> In-Reply-To: <902C948B-0A4C-48E1-8C6C-1BC7A15209D7@yahoo.com> From: Mateusz Guzik Date: Mon, 8 Dec 2025 13:46:45 +0100 X-Gm-Features: AQt7F2pg2uL5qFr3JtiXoSHOoxYosbCgbU54A4efWq1naQ6bmU9RfEMtzopOIbs Message-ID: Subject: Re: performance regressions in 15.0 [The Microsoft Dev Kit 2023 buildworld took about 6 minutes less time for jemalloc 5.3.0, not more, for non-debug contexts] To: Mark Millard Cc: Warner Losh , FreeBSD Current , FreeBSD-STABLE Mailing List , Konstantin Belousov Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dQ1vl5GS6z3Wkk On Sun, Dec 7, 2025 at 5:19=E2=80=AFPM Mark Millard wro= te: > > On Dec 6, 2025, at 19:03, Mark Millard wrote: > > > On Dec 6, 2025, at 14:25, Warner Losh wrote: > > > >> On Sat, Dec 6, 2025, 3:06=E2=80=AFPM Mark Millard = wrote: > >> > >>> On Dec 6, 2025, at 06:14, Mark Millard wrote: > >>> > >>>> Mateusz Guzik wrote on > >>>> Date: Sat, 06 Dec 2025 10:50:08 UTC : > >>>> > >>>>> I got pointed at phoronix: https://www.phoronix.com/review/freebsd-= 15-amd-epyc > >>>>> > >>>>> While I don't treat their results as gospel, a FreeBSD vs FreeBSD t= est > >>>>> showing a slowdown most definitely warrants a closer look. > >>>>> > >>>>> They observed slowdowns when using iperf over localhost and when co= mpiling llvm. > >>>>> > >>>>> I can confirm both problems and more. > >>>>> > >>>>> I found the profiling tooling for userspace to be broken again so I > >>>>> did not investigate much and I'm not going to dig into it further. > >>>>> > >>>>> Test box is AMD EPYC 9454 48-Core Processor, with the 2 systems > >>>>> running as 8 core vms under kvm. > >>>>> . . . > >>>> > >>>> > >>>> > >>>> Both of the below are from ampere3 (aarch64) instead, its > >>>> 2 most recent "bulk -a" runs that completed, elapsed times > >>>> shown for qt6-webengine-6.9.3 builds: > >>>> > >>>> 150releng-arm64-quarterly qt6-webengine-6.9.3 53:33:46 > >>>> 135arm64-default qt6-webengine-6.9.3 38:43:36 > >>>> > >>>> For reference: > >>>> > >>>> Host OSVERSION: 1600000 > >>>> Jail OSVERSION: 1500068 > >>>> > >>>> vs. > >>>> > >>>> Host OSVERSION: 1600000 > >>>> Jail OSVERSION: 1305000 > >>>> > >>>> The difference for the above is in the Jail's world builds, > >>>> not in the boot's (kernel+world) builds. > >>>> > >>>> > >>>> For reference: > >>>> > >>>> > >>>> https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D150re= leng-arm64-quarterly&build=3D88084f9163ae > >>>> > >>>> build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sun Nov 30= 05:40:02 -00 2025 > >>>> build time: 2D:05:33:52 > >>>> > >>>> > >>>> https://pkg-status.freebsd.org/ampere3/build.html?mastername=3D135ar= m64-default&build=3Df5384fe59be6 > >>>> > >>>> build of www/qt6-webengine | qt6-webengine-6.9.3 ended at Sat Nov 22= 15:33:34 -00 2025 > >>>> build time: 1D:14:43:41 > >>> > >>> > >>> Expanding the notes to before and after jemalloc 5.3.0 > >>> was merged to main: beefy18 was the main-amd64 builder > >>> before and somewhat after the jemalloc 5.3.0 merge from > >>> vendor branch: > >>> > >>> Before: p2650762431ca_s51affb7e971 261:29:13 building 36074 port-pack= ages, start 05 Aug 2025 01:10:59 GMT > >>> ( jemalloc 5.3.0 merge from ven= dor branch: 15 Aug 2025) > >>> After : p9652f95ce8e4_sb45a181a74c 428:49:20 building 36318 port-pack= ages, start 19 Aug 2025 01:30:33 GMT > >>> > >>> (The log files are long gone for port-packages built.) > >>> > >>> main-15 used a debug jail world but 15.0-RELEASE does not. > >>> > >>> I'm not aware of such a port-package builder context for a > >>> non-debug jail world before and after a jemalloc 5.3.0 merge. > >>> > >> A few months before I landed the jemalloc patches, i did 4 or 5 from d= irt buildworlds. The elasped time was, iirc, with 1 or 2%. Enough to see ma= ybe a diff with the small sample size, but not enough for ministat to trigg= er at 95%. I didn't recall keeping the data for this and can't find it now.= And I'm not even sure, in hindsight, I ran a good experiment. It might be = related, or not, but it would be easy enough for someone to setup a two jai= ls: one just before and one just after. Build from scratch the world (same = hash) on both. That would test it since you'd be holding all other variable= s constant. > >> > >> When we imported the tip of FreeBSD main at work, we didn't get a cpu = change trigger from our tests that I recall... > > > > > > The range of commits look like: > > > > =E2=80=A2 git: 9a7c512a6149 - main - ucred groups: restore a useful = comment Eric van Gyzen > > =E2=80=A2 git: bf6039f09a30 - main - jemalloc: Unthin contrib/jemall= oc Warner Losh > > =E2=80=A2 git: a0dfba697132 - main - jemalloc: Update jemalloc.xml.i= n per FreeBSD-diffs Warner Losh > > =E2=80=A2 git: 718b13ba6c5d - main - jemalloc: Add FreeBSD's updates= to jemalloc_preamble.h.in Warner Losh > > =E2=80=A2 git: 6371645df7b0 - main - jemalloc: Add JEMALLOC_PRIVATE_= NAMESPACE for the libc namespace Warner Losh > > =E2=80=A2 git: da260ab23f26 - main - jemalloc: Only replace _pthread= _mutex_init_calloc_cb in private namespace Warner Losh > > =E2=80=A2 git: c43cad871720 - main - jemalloc: Merge from jemalloc 5= .3.0 vendor branch Warner Losh > > =E2=80=A2 git: 69af14a57c9e - main - jemalloc: Note update in UPDATI= NG and RELNOTES Warner Losh > > > > I've started a build of a non-debug 9a7c512a6149 world > > to later create a chroot to do a test buildworld in. > > > > I'll also do a build of a non-debug 69af14a57c9e world > > to later create the other chroot to do a test > > buildworld in. > > > > non-debug means my use of: > > > > WITH_MALLOC_PRODUCTION=3D > > WITHOUT_ASSERT_DEBUG=3D > > WITHOUT_PTHREADS_ASSERTIONS=3D > > WITHOUT_LLVM_ASSERTIONS=3D > > > > I've used "env WITH_META_MODE=3D" as it cuts down on the > > volume and frequency of scrolling output. I'll do the > > same later. > > > > If there is anything you want controlled in a different > > way, let me know. > > > > The Windows Dev Kit 2023 is booted (world and kernel) > > with: > > > > # uname -apKU > > FreeBSD aarch64-main-pbase 16.0-CURRENT FreeBSD 16.0-CURRENT main-n2819= 22-4872b48b175c GENERIC-NODEBUG arm64 aarch64 1600004 1600004 > > > > which is from an official pkgbase distribution. So the > > boot-world is a debug world but the boot-kernel is not. > > > > The Windows Dev Kit 2023 will take some time for such > > -j8 builds and I may end up sleeping in the middle of > > the sequence someplace. So it may be a while before > > I've any comparison/contrast data to report. > > > > > Summary for jemalloc for before vs. at 5.3.0 > for *non-debug* contexts doing the buildworld : > > before 5.3.0: 9754 seconds (about 2.7 hrs) > with 5.3.0: 9384 seconds (about 2.6 hrs) > While in principle this can accurately reflect the difference, the benchmark itself is not valid as is. First, you can't just run it once -- the result needs to be proven repeatable and profiled. For a build of a that duration, for this few resources, for all I know the real factor was randomness from I/O. That aside you need a sanitized baseline. From the description it not clear to me at all if you are doing the build with the clang perf regression fixed or not. Even that aside, I outlined 3 more regressions: - slower binary startup to begin with - slower syscalls which fail with an error - slower syscall interface in the first place Out of the the first one is most important here. If I was to work on this, seeing that the question at hand is whether the jemalloc update is a problem, I would bypass all of the above and instead take 14.3 (not stable/14!) as a baseline + jemalloc update on top. This eliminates all of the factors other than jemalloc itself. building world also seems a little fishy here and it is not clear to me at all what version have you built -- was the new jemalloc thing building new jemalloc and old jemalloc building old jemalloc? More imporantly I would be worried some of the build picks up whatever jemalloc it finds to use during some of the build. I would benchmark this by building a big port (not timing dependencies of the port, just the port itself -- maybe even chromium or firefox). That's of course quite a bit of effort and if there is nobody to do that (or compatible), imo the pragmatic play is to revert the jemalloc update for the time being. This restores the known working state and should the update be a good thing it can land for 15.1, maybe fixed up.