From nobody Sun Oct 5 05:21:48 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cfW3z48pSz6BbjN for ; Sun, 05 Oct 2025 05:22:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-21.consmr.mail.gq1.yahoo.com (sonic313-21.consmr.mail.gq1.yahoo.com [98.137.65.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4cfW3y66rgz45vr for ; Sun, 05 Oct 2025 05:22:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=NVku18n8; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1759641720; bh=rS20WuHAESzakKhszOEzxOfifjBoxYWi+r7iVv0P/yo=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=NVku18n8vc/Ke1BOXi4urUNWKC7D3v5gSmzBDi4ij2o6bHpKub6eMODXwA7NhZm4v4FCdSQ3wckwD6rxjKH7NwSrp7lAjeZCVIHK/XQYQgEzq6T2kpqNqRVEVZm5hMiF3wZvh3SVJECdrbAy43OL6+Mkol/XGNEBpWV8j3tEo5RJcWFKYqJC+8rLnlpkMffR2kGkgWvaPc18TTLqAQQX+wrZxqUqHgFXuf8VTwfu0kfkMHBHGHSqSOkVpjaPj8f1PLE1IrDuQv9gyae8BgzDkI74kvNe679eFxdtnNiVJ2WRbjUog9Is77+66Pb7lNqovFRwi8MOJ2oMpNwglP6fEw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1759641720; bh=SJtTEaCLu4fcv7NzK7iq6duXkCajNa3mrM6gEz7hz4v=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Gf94QlL1v44YqD7eZZhxYjG+Mb/l4woFBkjlzBQJBdypV2Mzw4iSmIEncInrdAQtwIE4xfhtWCgQxvBsgihCjg5LMG9B2t3V8OxAQbG8hklu5JNnhmf8jDguLNtsekIuj048RAbBZKFq7sigTuxZ/JsGMJlInu9xQUb346Mu0J2PwoDHh1GLlTqH1gbuHDJsrcahg6vRxyyC3LphHvnhRHasmMHus2UkyaxgGKoxNSWTYjpi90yvN6jjPB9jVhONthWQnkH4YEmdII3SP7wOoTtr1Vx9odAJ2awMdiy5Y4gkYopxoAqMFUtUI24yPjGg/RrRYLzmGR8xzLPglKC3Lw== X-YMail-OSG: N0g7c88VM1mv5LZI64YKyDEro7wuLM87Zd_SQMJiz7GJWARhQSnH5RYgtZNLwW6 D120Xig6W5JoMLYFLWirAHpd4J2Mx3QHoiY8xbZanZz6Ze.PIltbfu5T2l.sR_zRhyDZHde.NVPT vVQ3TavDed3nfFWrQqexszAZNUmacZiAyyCVitAcELhU3EPPjKL4E073ImyLNV5Ua6smxL5k4Lxz EEP4wthA4TxX_Zd8dDS6WE8QpgVYYqI7LpYOW0pOTJK.JTMqRJ432bLKProwlKRtSOVBOgIURUIH akKt2kq0D7kKuuSElX.pxAZK90Xl.Vc6PT3v9BbTt4sJs0oX_IgIUE5nQNwW4xWQyKAvVQQ4KjOR 7RF10JjEAmkFyXH72Rxp2QuRtM.VB4SR0JvFFnoC5kqG1URJCoLrlfkZ.f3lnpo_z5_W36t80ytL DodHvrJMr5w8rPARlqs4UTwOr.KlYVbg.cg6xqxqoNhsge5SEPWGmqQgt.Up1oFppMAm5N.us_Is Lt44OnbKHXT0nbyKzT4pcdUqNJlwf0P1u1OUawVvf9mDxlLglPrWaZu8jrApw4Bkos.RHyxmMwC_ 295LPPZ76.Am8cy5ISCpt0h7ax_BDBezsf7Fxly8ylVS5cqufkgCZo587asDE1xOYLGLY865mLHT RIAJ5mjbI4YxgV02ECNY.hI.QS2sXCQWH6jdI8FRUjRL3XjGbmSEWu1CcpsRPDB8jeTPyAP5FGud bu9HxI7g25xLvahNMch4katS6Cq.A9mUpI0JDZa0.hDrHifNN28.7hYyC31dDJLBJYvirnZMUwtz njwWOatp0lShUgo1ncXqC2rpQqWuTIamxCWJXseNVvNlCBIvmqm1dm.KsBYBSUEngniv6jDmwt3W rBGjUE9qhn1JR5bLZMi7Bay5bykU4a4BYdcLk0E0rImRCs_Ni0BQT.pS.HOybofeG10SRXB2lOdN gWfKUDUfRo0l0EzeWPGCb0Pv7jO2tThDHQHBMmSPlebsPRXYieX5fie5zigt14k4zTqLYHjBbR7K 3NkUuUJbG9v6DE2UkODTZ154GcpjznSvkMjqseeP18E5qTyJx3rZBgcYVPWDSDLGjZtSsflKbNL8 nnglwXbZnaWi.XCV22C1gNNytptJ615gic9XLiH4WmO9j5UvukdY81VPbEVMyK6ysrtkr4es3PXN jJYSSpHzblWtEvl._hxOogPbkcndC1OxIjqmqHimib8HS6m8iCnIH4ozKzipVuXBcoAQ.uwfHchq r4x.qKiV0diMAbWRcNklKKRGC1Isy7CGV1cHMAI8k3WIwsahX0s4f5qw1dfOOrmzeacK8Ji7QwjH aee_FQFlrjIDzEpJHv_HLLw96GvV9ZXlgGr7KIu3m9J_r9.eK0Xsr2u9CzdTanO7sWSbVECOszA7 EmMt9v_Ij8WCIzoEkb3fDC78MwzOKgDx.VXbJq.d776pBXxrCpEKpmSW5U8KVV8Hwa.Aobpkndf4 IhRrOZ1MwnaRzeItth52Gg7KD9WzTWB2uYHgzu.3SSlex3uUsUX00fZN530ciK0I97SYVvel2hZE AQW2MnklVBX69OWtOUryW07UVnkpAo2C82h7.Gt.CzJJxz9x_xCYkM0eJmRSDR9zT6VFqcSEPIo1 0md6zbVg16MYyzDRmmYPPeI3Zn1XrhOi6Qgan_dQOSq3yZWfo04DHqR5NGgPdmHAAzMQZGCVBiCN jzryMAJlKHVvuEoJQ.iG7X75Hsc95RRKo3RZSDWa.AjEjmytRuDEe_zz4IucvZ8Lxm__REVwTr_1 viZrInI_bkAgTHyN4ynAuTfO_o_lVoHcjg4IHZWNE3Wbc3JWY_HDydeNKGqpzISM4L3rUd1mkqTM u1rYPIi1ZpIBlfGD8N9M7SNoFDq.Pp2Hst2SwGv28q4NK0DINMCGaU9I7zvU2320xqKyRz6BOrrG loZM5I2r6JrnZZ66D4q_l_QlwmRWL76N5IGoeRMj25LQaFjUuDNrNLN52vgXHkjKUTtet5p1vpSK _WrMWuSLDFRaODV.zDHTmpiZN0uz3zx3eVZ4mF5t0MCjJLhq7fvuOgOLNN7q2z8VCRgRPu2G_57H cP6rp6_vZmTb41pOUY.kr8mb9mJKMRcMms8tzcAylXdZOTUtg_PTXWPMtegvvxChwZMd6XXLnFIj 3YSyKeZ3FBUuc7u2.jE70R0e5sUAgGYR11qSelbUx_6Lf0xpMF_MlWgliBhGvxFz7Ig82_GN93WJ ieYiEG9.CJoXOv7O5HjhAMluI2teWu_.g6R2kWsQlbgvkLIMwD6UPqEHrxY2MgDjKZ3FDt0aPfV1 USw-- X-Sonic-MF: X-Sonic-ID: 0eac3319-29a3-4410-a1bd-eda0b4cc0f38 Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Sun, 5 Oct 2025 05:22:00 +0000 Received: by hermes--production-gq1-66b66ffd5-r958h (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID fca2409c47e329d5facab27dd3e6164a; Sun, 05 Oct 2025 05:21:59 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: main 16 and 15.0-ALPHA4 [on amd64]: using a USB3 context gets extensive "flswai" [and "rename"] STATE time during poudriere builds (UFS context happens to be in use); more Date: Sat, 4 Oct 2025 22:21:48 -0700 References: <46E0F6E8-A365-4C01-BFF8-CE2423B6DA00@yahoo.com> <4D74D446-2078-4A5F-B245-913273E2DDD1@yahoo.com> <03B6ABBE-DF0B-441F-9ABD-5565ADBF29ED@yahoo.com> <93873756-54C8-4408-9395-5E67F0300D5E@yahoo.com> <9204210A-E61D-4C5D-823B-878848698847@yahoo.com> To: FreeBSD Current , FreeBSD-STABLE Mailing List In-Reply-To: <9204210A-E61D-4C5D-823B-878848698847@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.86 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; NEURAL_HAM_SHORT(-0.86)[-0.861]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FREEMAIL_FROM(0.00)[yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.84:from]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.84:from] X-Rspamd-Queue-Id: 4cfW3y66rgz45vr On Oct 2, 2025, at 21:43, Mark Millard wrote: > On Oct 2, 2025, at 20:45, Mark Millard wrote: >=20 >> On Sep 30, 2025, at 20:43, Mark Millard wrote: >>=20 >>> [The new material here ends up being about nameicap_cleanup >>> and its exclusive use of mnt_renamelock being one potential >>> bottleneck involved here. I make no claim it has anything to >>> do with the flswai activity reported. The possible >>> bottleneck is an observation, not something that I claim >>> there is any alternative to. I do not know if this is of any >>> interest or not.] >>>=20 >>> On Sep 29, 2025, at 16:06, Mark Millard wrote: >>>=20 >>>> On Sep 29, 2025, at 13:01, Mark Millard wrote: >>>>=20 >>>>> An example is during the cpdup activities when multiple happen >>>>> in overlappingtime frames: >>>>=20 >>>> I'll note that I see this on the amd64 32-FreeBSD-cpu system >>>> but not on the aarch64 8-FreeBSD-cpu Windows Dev Kit 2023 >>>> system. May be at some point I'll try the older 16-FreeBSD-cpu >>>> aarch64 (Cortex-A72) system. >>>>=20 >>>> Also, on the 7950X3D amd74 system, I see the behavior with >>>> 14.3-Stable. Apparently, this is not new with 15+. It has >>>> been a long time since I'd tried using an amd64 system for >>>> such activity based on using USB3 media. But it has been >>>> common for me for aarch64 over that time frame. >>>>=20 >>>>> . . . >>>>> . . . >>>=20 >>> . . . >>>=20 >>>=20 >>>>> . . . >>>>> . . . >>>>>=20 >>>>> But I'll also see such on c compiles, ld commands, etc. I've >>>>> not seen rename for pkg-static but I have seen flswai for it. >>>>>=20 >>>>> The system spends lots of time 95%+ idle from the wait >>>>> activities. >>>>>=20 >>>>> I see such directly booted from the USB3 media (a 15.0-ALPHA4 >>>>> context on UFS media) and when using that media via chroot >>>>> from both ZFS and UFS boots that are not USB based. The ZFS >>>>> and UFS boots do not show the behavior with the normal >>>>> non-USB3 media used instead. >>>>>=20 >>>>> The system in use is an AMD 7950X3D with 32 FreeBSD cpus, >>>>> 192 GiBytes of RAM. main 16 booting for non-USB boots >>>>> and 15.0-ALPHA4 boots for the USB3 boots. kernel and >>>>> world are via official pkgbase distribution installs: >>>>> it is not a personal build of the kernel or world. >>>>>=20 >>>>>=20 >>>>> . . . >>>>> . . . >>=20 >> I got a test context were I could compare the same >> media used on the same USB4 port on a laptop >> (Dell Precision 5490, 22 FreeBSD cpus, 32 GiBytes >> RAM, 4 USB4 ports), where, on boot, the media ends >> up being handled as either: >>=20 >> ) "nda0 at nvme0" (via involving a Thunderbolt 3 hub) >> ) "da0" (via a direct connection) >>=20 >> (The UEFI/ACPI does enough to make basic operation >> work, presenting some view to FreeBSD for media on >> USB4 ports.) >>=20 >> Both have the bottlenecks visible when monitored with >> top, both "flswai" and "rename" examples occur in >> both contexts. >>=20 >> But "nda0 at nvme0" bottleneck periods do not last >> nearly as long as "da0" bottleneck periods do, making >> "nda0 at nvme0" use much more reasonable for the >> type of activity. >>=20 >> Still, this eliminates the possibility that the issue >> was limited to USB. It also eliminates it being >> specific to the prior AMD (7950X3D) test context. >>=20 >>=20 >> For reference: >>=20 >> # uname -apKU >> FreeBSD USB4sys 16.0-CURRENT FreeBSD 16.0-CURRENT = main-n280801-213170eb956f GENERIC-NODEBUG amd64 amd64 1600001 1600001 >>=20 >> (It is from official pkgbase distribution use, a >> copy of another boot media with some parameters >> replaced afterwards.) >>=20 >> QUOTE >> CPU: Intel(R) Core(TM) Ultra 7 165H (3072.00-MHz K8-class CPU) >> Origin=3D"GenuineIntel" Id=3D0xa06a4 Family=3D0x6 Model=3D0xaa = Stepping=3D4 >> . . . >> WARNING: L3 data cache covers more APIC IDs than a package (6 > 3) >> FreeBSD/SMP: Multiprocessor System Detected: 22 CPUs >> FreeBSD/SMP: Non-uniform topology >> END QUOTE >>=20 >> (The internal NVMe media has Dell's ubuntu on it.) >>=20 >>=20 >> Note: >> Ignoring very old Intel MacBook Pro's and Mac Mini >> 2018's, none of which I've ever native-booted FreeBSD >> on, the Dell P. 5490 is the only Thunderbolt or USB4 >> based system that I've access to. >=20 > I've discovered what is tied to the flswai/rename: > Soft Updates had been enabled. Disabling Soft Updates > leads to biowr and getblk as what shows for the > bottleneck. (There is still a bottleneck.) >=20 An FYI showing what can happen for the 32 FreeBSD cpu USB3-based test context. The below occurred later, after the initial bottleneck sequence when the jails are being filling in via cpdup of the ref (for UFS context via chrooting into it to then start the build): # uname -apKU FreeBSD 7950X3D-ZFS 16.0-CURRENT FreeBSD 16.0-CURRENT = main-n280910-85531add2844 GENERIC-NODEBUG amd64 amd64 1600001 1500066 [00:18:54] [official-amd64-default] [2025-10-04_21h23m47s] = [parallel_build] Time: 00:17:58 Queued: 693 Inspected: 0 Ignored: 0 Built: 33 Failed: 0 = Skipped: 0 Fetched: 0 Remaining: 660 ID TOTAL ORIGIN PKGNAME = PHASE TIME TMPFS CPU% MEM% [29] 00:08:55 databases/lmdb | lmdb-0.9.33,1 = starting 00:08:55 =20 [01] 00:02:53 graphics/libpotrace | libpotrace-1.16 = starting 00:02:53 =20 [15] 00:07:37 archivers/lzo2 | lzo2-2.10_1 = starting 00:07:37 =20 [30] 00:07:22 lang/lua53 | lua53-5.3.6_1 = starting 00:07:22 =20 [02] 00:07:51 devel/libmtdev | libmtdev-1.1.7 = starting 00:07:51 =20 [16] 00:07:35 lang/lua54 | lua54-5.4.8 = starting 00:07:35 =20 [31] 00:07:27 sysutils/libsunacl | libsunacl-1.0.1_1 = starting 00:07:27 =20 [03] 00:08:07 sysutils/dmidecode | dmidecode-3.6 = starting 00:08:07 =20 [17] 00:08:33 x11/xorgproto | xorgproto-2024.1 = starting 00:08:33 =20 [32] 00:07:38 print/indexinfo | indexinfo-0.3.1_1 = starting 00:07:38 =20 [04] 00:06:18 devel/libdatrie | libdatrie-0.2.13_2 = starting 00:06:18 =20 [18] 00:07:52 shells/bash-completion-freebsd | = bash-completion-freebsd-1.4.0 starting 00:07:52 =20 [05] 00:08:22 audio/libvorbis | libvorbis-1.3.7_2,3 = starting 00:08:22 =20 [19] 00:07:34 comms/iwmbt-firmware | iwmbt-firmware-20250410 = starting 00:07:34 =20 [06] 00:04:56 sysutils/sdparm | sdparm-1.12_1 = starting 00:04:56 =20 [20] 00:08:01 shells/ksh93 | ksh93-93.u_4,2 = starting 00:08:01 =20 [07] 00:08:02 textproc/sdocbook-xml | sdocbook-xml-1.1_2,2 = starting 00:08:02 =20 [21] 00:08:49 archivers/libmspack | libmspack-0.11alpha = starting 00:08:49 =20 [08] 00:08:23 benchmarks/iperf3 | iperf3-3.19.1 = starting 00:08:23 =20 [22] 00:08:01 textproc/xmlcharent | xmlcharent-0.3_2 = starting 00:08:01 =20 [09] 00:08:03 devel/pkgconf | pkgconf-2.4.3,1 = starting 00:08:03 =20 [23] 00:08:15 lang/perl5.42 | perl5-5.42.0_1 = starting 00:08:15 =20 [10] 00:08:16 devel/opencl | opencl-3.0.19 = starting 00:08:16 =20 [24] 00:08:11 benchmarks/bonnie | bonnie-2.0.6_2 = starting 00:08:11 =20 [11] 00:08:45 security/easy-rsa | easy-rsa-3.2.4,1 = starting 00:08:45 =20 [25] 00:07:48 audio/mpg123 | mpg123-1.33.2 = starting 00:07:48 =20 [12] 00:03:34 databases/sqlite3@default | sqlite3-3.50.2_1,1 = starting 00:03:34 =20 [26] 00:08:45 ports-mgmt/portconfig | portconfig-0.6.2_2 = starting 00:08:45 =20 [13] 00:07:54 textproc/expat2 | expat-2.7.3 = starting 00:07:54 =20 [27] 00:07:49 multimedia/v4l_compat | v4l_compat-1.23.0_7 = starting 00:07:49 =20 [14] 00:08:27 dns/public_suffix_list | = public_suffix_list-20250828 starting 00:08:27 =20 [28] 00:07:54 devel/libdwarf | libdwarf-20161124 = starting 00:07:54 =20 It does eventually make progress again after this. The laptop 22 FreeBSD cpu context with external USB4 media being handled as "nda0 at nvme0" (via using a Thunderbolt 3 hub between the laptop and the media) has much better performance for builds. (No chroot is involved here.) # uname -apKU FreeBSD USB4sys 16.0-CURRENT FreeBSD 16.0-CURRENT = main-n280910-85531add2844 GENERIC-NODEBUG amd64 amd64 1600001 1600001 Shorter duration bottlenecking is still visible but use wold be much more reasonable in this type of context. I''ll note that I did not notice any "rename" STATE bottleneck time during this activity but did see "flswai" STATE bottlenecking. (Soft Updates was enabled.) =3D=3D=3D Mark Millard marklmi at yahoo.com