From nobody Sat Feb 4 15:16:17 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P8GM765TBz3nW5l for ; Sat, 4 Feb 2023 15:16:19 +0000 (UTC) (envelope-from debdrup@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P8GM75fRcz3wNJ for ; Sat, 4 Feb 2023 15:16:19 +0000 (UTC) (envelope-from debdrup@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675523779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ZXeHEBriOVyVhW83egLaK4xy3Jmz97FPB6mDYgWH8RE=; b=E/Wuo+PgdtjZ13HoYGqTMzg+yRS8yTq0CRdwG7HeHWtGidgKcVvydMk0RbPf+Pw70kbGBj EpNNfx6IYQdkanYcWc3YTnGyWyufeTgcINMIMgmWcq2qiSgFEBopdnNvHyYYJQa+Zymnoo 8FIu6t3jXZjUXZlyDVQS9274HEka0SMMa2Y5ZAmwzx7xtf5xHDng7QMtG8NG4Wi8rOpeEe Li6FQDxMFQyp8Qlgyf46WTfUz7xg+d7vPXHzzuNBdbG5Rjqj4N9ljCMDGRuANoPKaXpINM d4LWQeWMP20ZZt3irSYvBOThs23MuBjt/w95cBXPu9pld7pOcolgerbEVkcOFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675523779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ZXeHEBriOVyVhW83egLaK4xy3Jmz97FPB6mDYgWH8RE=; b=GUXS6p5AQmRpo2VOCmkZ5OiXEu7jnvxzP21T/bIFyiWwi5kKpuXdxKWK7+NU9wOMgm6bfz TQ82rBHtCxkvQHL8aRguMGw3B5BnC9KFtAFk3wl/uzxFvOyaGYTSMMEe+P1P34IIEb5WlO PYHQ2PZnNjSs4uuboNDbdmYnkmUTUkdVhIjVcikxtOSRwe5LtqebdzHCmpJ/4GivxcGzEd xgr6imEblCdRU4B+UIuQIy6VB+/sqoRlklNxmDOhtkum+59zcUMJTpoN+UcNzGbYzthzka T2dTJpDRH3RZdPuN/JYVIxLsDX5shTVus5JqbcyFV6IHN4IOb+8h2NB4PG664w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675523779; a=rsa-sha256; cv=none; b=A/FyxZf92aAAqF7j+16gTn5a4JinhfkYrNaT+I5HcUQeF4TrCqCtEVIK/gf0zRLPvwMJRu HqGD5vS9o5PidiOsqLsJAlEe1Y5YOHURCXNfK7XQhro+YdBbnKLeiUDNM3kXKs9cvSz/tX tCPwgwQilllLvLhptfQv0cV264ZtOxIc8sy2ReDX/Lwd9MGQOa+x291nu+AvGyhIjx3J76 okcY0shNAxsuNJz+7mi3BU2eqlK0fMPdYrrlUA44iXDFIYmzk78MzbLCXru97R75TbnYB3 ywOKlwkLRmIAw4/O4HDNLIX5Cab6fzkTF0GudmlfC3s+k3vd3eq8kA4O7M4adg== Received: by freefall.freebsd.org (Postfix, from userid 1471) id B0458122A3; Sat, 4 Feb 2023 15:16:19 +0000 (UTC) Date: Sat, 4 Feb 2023 16:16:17 +0100 From: Daniel Ebdrup Jensen To: freebsd-hackers@freebsd.org Subject: Re: Swap, ZFS & ARC Message-ID: <20230204151617.wlan22x5jvae5jzm@geroi.local> References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="n5tann3f7es4la3y" Content-Disposition: inline In-Reply-To: X-ThisMailContainsUnwantedMimeParts: N --n5tann3f7es4la3y Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline On Thu, Feb 02, 2023 at 02:28:48PM +0000, jbo@insane.engineer wrote: >Hello folks, > >Based on a discussion on the forums not so long ago I tried to figure out how swap usage on a ZFS system plays together with ARC. However, I could find very little to no information on this which leads me to believe that there is some "core concept" I might be oblivious to. > >The main question is basically this: Your system starts to swap out data from RAM to your swap partition. This swap data on disk ultimately resides somewhere in a ZFS pool. If this data then gets accessed, it might be cached by ARC essentially eating up memory again which seems counter productive. >Is there any magic which prevents swap partitions from being loaded into ARC? Or is this a non-issue for some other reason? > >Best regards, >~ joel Hi Joel, The catch-22 mentioned elsewhere in the thread isn't unique, it's a function of how virtual memory systems work when they have paging, even if they're designed to try and work around them (which, as others have pointed out, is how FreeBSD works). The only way around it that I know of is to adjust the VM watermarks that control when certain triggers happen, but unfortunately this process is workload dependent, so there's no real way of making recommendations short of "if you're still seeing the catch-22 after adjusting the values, adjust them more". The values that need to be adjusted are the following OIDs, using sysctl(8): vm.v_free_min, vm.v_free_target, vm.v_free_reserved, vm.v_inactive_target, and vm.v_free_severe. There is a pretty big downside to this, though - which is that it means a much bigger portion of your memory will be completely unused at any given point; which consequently mean your ARC is much less efficient, there's less room for more things in other caches, and you're still paying for the electricity to keep the memory state of the unused memory. There's also an issue you can run into if you ever have a panic(9) and it's caused by something in ZFS, because that'll mean that the core file can't be reliably expected to produce a meaningful backtrace. Ultimately, it's a question of what's easier - if you've setup your pool with no swap, and didn't reserve some space for future expansion and/or can't be arsed to do the zfs send | receive onto a new pool that has the space for it, the above method will at least let you get swap on ZFS without having to worry about the ZPL from using file extents (which is never a good idea, not even on UFS). As for your second question, the dataset zfsprops(7) parameter that you're looking for is primarycache, as it can be set to none. You'll also want to use org.freebsd:swap, turn off checksumming and compression, and disable sync. Yours, Daniel Ebdrup Jensen --n5tann3f7es4la3y Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGTBAABCgB9FiEEDonNJPbg/JLIMoS6Ps5hSHzN87oFAmPedsFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDBF ODlDRDI0RjZFMEZDOTJDODMyODRCQTNFQ0U2MTQ4N0NDREYzQkEACgkQPs5hSHzN 87oEIAf/aXiuKuytzfTqcLUaVppewhFx1DFM2jYpPCW5LQNGEHlYP7nEwzoMLhN8 ACBLer8H//d9C2x1znTGRmeEhtnOwnpyEu1TInbRhiUJ21DufDzQ5OHzTI7ztxWk eVde3WtcZ0mdEShDpCUASAkRexA18ycd/t5YXqjgsik8TiZOfVVDMLvinxmPBelv /UC9gX8cdUnr2hF8e3UytP9JTpnAVahFvlxtTsiHEKk4ViPaTWwmakMR/npk7czW ff1omhndZt/RKvJOhVA/myV1TugNYo1HfRGj5QPlSMbsum2B0CM4oItaTEs1Ava2 eby1q2V24xSavDJnAqa9YA6eesFMhg== =P653 -----END PGP SIGNATURE----- --n5tann3f7es4la3y-- From nobody Tue Feb 7 06:02:23 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P9t4R2mYLz3pD9y for ; Tue, 7 Feb 2023 06:09:11 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay08.ispgateway.de (smtprelay08.ispgateway.de [134.119.228.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4P9t4P6npbz42kc for ; Tue, 7 Feb 2023 06:09:09 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of freebsd-listen@fabiankeil.de has no SPF policy when checking 134.119.228.98) smtp.mailfrom=freebsd-listen@fabiankeil.de; dmarc=none Received: from [91.20.76.172] (helo=fabiankeil.de) by smtprelay08.ispgateway.de with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pPHAB-0004Fr-4b for freebsd-hackers@freebsd.org; Tue, 07 Feb 2023 07:09:07 +0100 Date: Tue, 7 Feb 2023 07:02:23 +0100 From: Fabian Keil To: freebsd-hackers@freebsd.org Subject: Re: ZFS-related panic(s) with zfs-2.1.7-FreeBSD_g21bd76613? Message-ID: <20230207070223.1195e73b@fabiankeil.de> In-Reply-To: <20230125111011.455923bf@fabiankeil.de> References: <20230107174159.1b7e61e9@fabiankeil.de> <20230125111011.455923bf@fabiankeil.de> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/I=o8=qUc/1zYY0gg+IqzxTX"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Df-Sender: Nzc1MDY3 X-Spamd-Result: default: False [-3.18 / 15.00]; SIGNED_PGP(-2.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.98)[-0.983]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[134.119.228.98:from]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:34011, ipnet:134.119.228.0/24, country:DE]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[134.119.228.98:from]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; R_SPF_NA(0.00)[no SPF record]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[fabiankeil.de]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4P9t4P6npbz42kc X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --Sig_/I=o8=qUc/1zYY0gg+IqzxTX Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Fabian Keil wrote on 2023-01-25 at 11:10:11: > Fabian Keil wrote on 2023-01-07 at 17:41:5= 9: >=20 > > Yesterday I rebased ElectroBSD [0] on stable/13 77c0992af4e3b > > while it was previously based on stable/13 d3b97a1ea0123. > >=20 > > I didn't notice any issues in a test VM and therefore decided > > to update my laptop as well. > >=20 > > So far I've experienced three panics/reboots/freezes that I suspect > > might be caused by the upgrade from zfs-2.1.6-FreeBSD_g6a6bd4939 > > to zfs-2.1.7-FreeBSD_g21bd76613. > >=20 > > They all occurred while I was syncing ZFS datasets with zogftw [0]. > >=20 > > Unfortunately I only have one backtrace so I can't say for > > sure that the other times where ZFS related as well: > >=20 > > Unread portion of the kernel message buffer: > > panic: VERIFY3(0 =3D=3D zap_remove(mos, dsobj, spa_feature_table[f].fi_= guid, tx)) failed (0 =3D=3D 2) > >=20 > > cpuid =3D 3 > > time =3D 1673033419 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00d= c6868a0 > > vpanic() at vpanic+0x151/frame 0xfffffe00dc6868f0 > > spl_panic() at spl_panic+0x3a/frame 0xfffffe00dc686950 > > dsl_dataset_deactivate_feature_impl() at dsl_dataset_deactivate_feature= _impl+0xe6/frame 0xfffffe00dc6869a0 > > dsl_dataset_clone_swap_sync_impl() at dsl_dataset_clone_swap_sync_impl+= 0x135/frame 0xfffffe00dc686ad0 > > dmu_recv_end_sync() at dmu_recv_end_sync+0x2a2/frame 0xfffffe00dc686b30 > > dsl_sync_task_sync() at dsl_sync_task_sync+0xb4/frame 0xfffffe00dc686b60 > > dsl_pool_sync() at dsl_pool_sync+0x42b/frame 0xfffffe00dc686be0 > > spa_sync() at spa_sync+0xb00/frame 0xfffffe00dc686e10 > > txg_sync_thread() at txg_sync_thread+0x281/frame 0xfffffe00dc686ef0 > > fork_exit() at fork_exit+0x7e/frame 0xfffffe00dc686f30 > > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00dc686f30 > > --- trap 0x3, rip =3D 0xffffffff80659b3f, rsp =3D 0, rbp =3D 0xffffffff= 818f4fa0 --- > > mi_startup() at mi_startup+0xdf/frame 0xffffffff818f4fa0 > > swapper() at swapper+0x69/frame 0xffffffff818f4ff0 > > btext() at btext+0x22 > > Uptime: 6m35s > > Dumping 1098 out of 8050 MB:..2%..11%..21%..31%..41%..51%..62%..72%..81= %..91% > >=20 > > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > > 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(struct pcpu, > > (kgdb) where > > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > > #1 dump_savectx () at /usr/src/sys/kern/kern_shutdown.c:394 > > #2 0xffffffff806cda18 in dumpsys (di=3D0x0) at /usr/src/sys/x86/includ= e/dump.h:87 > > #3 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:423 > > #4 kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:497 > > #5 0xffffffff806cde9e in vpanic (fmt=3D, ap=3Dap@entry= =3D0xfffffe00dc686930) at /usr/src/sys/kern/kern_shutdown.c:930 > > #6 0xffffffff81278e3a in spl_panic (file=3D, func=3D, line=3D, fmt=3D) at /usr/src/sys/co= ntrib/openzfs/module/os/freebsd/spl/spl_misc.c:107 > > #7 0xffffffff813001e6 in dsl_dataset_deactivate_feature_impl (ds=3Dds@= entry=3D0xfffff80019a60000, f=3Df@entry=3DSPA_FEATURE_USEROBJ_ACCOUNTING, t= x=3Dtx@entry=3D0xfffff80191ace200) > > at /usr/src/sys/contrib/openzfs/module/zfs/dsl_dataset.c:1116 > > #8 0xffffffff81304cb5 in dsl_dataset_clone_swap_sync_impl (clone=3D0xf= ffff8018fc79000, origin_head=3D, tx=3D, tx@entry= =3D0xfffff80191ace200) at /usr/src/sys/contrib/openzfs/module/zfs/dsl_datas= et.c:4083 > > #9 0xffffffff812eaff2 in dmu_recv_end_sync (arg=3D0xfffffe00d91366b8, = tx=3D0xfffff80191ace200) at /usr/src/sys/contrib/openzfs/module/zfs/dmu_rec= v.c:3233 > > #10 0xffffffff8132c254 in dsl_sync_task_sync (dst=3D0xfffffe00d91364a8,= tx=3Dtx@entry=3D0xfffff80191ace200) at /usr/src/sys/contrib/openzfs/module= /zfs/dsl_synctask.c:248 > > #11 0xffffffff8131ea6b in dsl_pool_sync (dp=3Ddp@entry=3D0xfffff801eaba= d800, txg=3Dtxg@entry=3D3576757) at /usr/src/sys/contrib/openzfs/module/zfs= /dsl_pool.c:847 > > #12 0xffffffff81353930 in spa_sync_iterate_to_convergence (spa=3D0xffff= fe00da149000, tx=3D0xfffff80191e73400) at /usr/src/sys/contrib/openzfs/modu= le/zfs/spa.c:9069 > > #13 spa_sync (spa=3Dspa@entry=3D0xfffffe00da149000, txg=3Dtxg@entry=3D3= 576757) at /usr/src/sys/contrib/openzfs/module/zfs/spa.c:9287 > > #14 0xffffffff81368281 in txg_sync_thread (arg=3Darg@entry=3D0xfffff801= eabad800) at /usr/src/sys/contrib/openzfs/module/zfs/txg.c:591 > > #15 0xffffffff80689fde in fork_exit (callout=3D0xffffffff81368000 , arg=3D0xfffff801eabad800, frame=3D0xfffffe00dc686f40) at /usr= /src/sys/kern/kern_fork.c:1093 > > #16 > > #17 mi_startup () at /usr/src/sys/kern/init_main.c:322 > > #18 0xffffffff80a1e439 in swapper () at /usr/src/sys/vm/vm_swapout.c:755 > > #19 0xffffffff802f8722 in btext () at /usr/src/sys/amd64/amd64/locore.S= :80 > >=20 > > Has anyone else seen this? > >=20 > > I've seen it with three different ZFS pools and I think the pools > > are fine. The laptop only supports USB2 so scrubbing the pools takes > > days which is why I didn't do it yet. > >=20 > > I have no reliable way to reproduce the issue yet. > > Running zogftw sync again after rebooting worked in all three cases. > [...] > > [0] > > [1] >=20 > I'm still getting panics with the stack trace above with > various pools about once a day on my work laptop and am > considering reverting the ZFS-related commits. Just for the archive: A couple of days ago I rebased ElectroBSD on my laptop on stable/13 20cfc902d911a and as a result I'm now using zfs-2.1.9-FreeBSD_g92e0d9d18 and zfs-kmod-2.1.9-FreeBSD_g92e0d9d18 there. With this combination I haven't seen any ZFS-related panics yet even though I syncronized a bunch of different ZFS pools with zogftw. Fabian --Sig_/I=o8=qUc/1zYY0gg+IqzxTX Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQTKUNd6H/m3+ByGULIFiohV/3dUnQUCY+HpcAAKCRAFiohV/3dU nZwdAJ9+t+sVxz685ka+NAmcODnmSGS0SQCfQrepK9wZVr1emxlmwhDG5+RQjLg= =7FSy -----END PGP SIGNATURE----- --Sig_/I=o8=qUc/1zYY0gg+IqzxTX-- From nobody Tue Feb 7 23:30:43 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PBKBV0mJ6z3nrlv for ; Tue, 7 Feb 2023 23:30:58 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ua1-f45.google.com (mail-ua1-f45.google.com [209.85.222.45]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PBKBT1yKBz3sCP for ; Tue, 7 Feb 2023 23:30:57 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.222.45 as permitted sender) smtp.mailfrom=asomers@gmail.com; dmarc=none Received: by mail-ua1-f45.google.com with SMTP id b18so3076630uan.11 for ; Tue, 07 Feb 2023 15:30:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9PF90ic+mQSSwesONnXBJVRHNM4+pAvMJ47WhfGldM0=; b=bKRX/+SqMjLHXknQh8RX25o2riJPyQ63synhefLlMWScMNB88Y0JfgQPybIwKzARSc 7WMGujSkPYoh8tNQOaUBsO3ENGHgWHEpJ8KJRL9akRd+imdiBUWAH26m1Tpoq2u0mlym 2d+JPzvZupJ+93SrWVefNZIQnK2RWhkfTWIaLZvhh89YzKy17+kQm8EFQ4ks2OI9VcmF rxmZjoMTA7rcHziYcUR7odzbsGH7CbJt1w+lkGzPlxMQSJXoY/1hGHV2yvOnZcY+7tWe IdX6OIegbQ5pvRBomlxrVwufdjyzz1WYWoJE1SE5/LYnThJ3VpAPo27MmQiFxgQI4yMl 6pXA== X-Gm-Message-State: AO0yUKVkPWZfKF81VofGnmJjVxFDuibik8/ASspxcR8HUnjC7jpUVLzD VPEGiIaTY2ZR/O0+zuHj8NVv6PZ6gvBGkdW9CNcTyvJd X-Google-Smtp-Source: AK7set8lOwgV3he8Px00r0VFtyPqllSksnxWSotTZQ9IuLpgPfWz+Yb6K5hGq3j0hxKM+S4i+vgCP/LLqWwOoZjZufg= X-Received: by 2002:ab0:6644:0:b0:65a:ed38:427a with SMTP id b4-20020ab06644000000b0065aed38427amr1185471uaq.21.1675812655411; Tue, 07 Feb 2023 15:30:55 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Tue, 7 Feb 2023 16:30:43 -0700 Message-ID: Subject: Fwd: RFC: GEOM and hard disk LEDs To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-2.93 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.93)[-0.934]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[209.85.222.45:from]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.222.45:from]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[asomers]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Rspamd-Queue-Id: 4PBKBT1yKBz3sCP X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Most modern SES backplanes have two LEDs per hard disk. There's a "fault" LED and a "locate" LED. You can control either one with sesutil(8) or, with a little more work, sg_ses from sysutils/sg3_utils. They're very handy for tasks like replacing a failed disk, especially in large enclosures. However, there isn't any way to automatically control them. It would be very convenient if, for example, zfsd(8) could do it. Basically, it would just set the fault LED for any disk that has been kicked out of a ZFS pool, and clear it for any disk that is healthy or is being resilvered. But zfsd does not do that. Instead, users' only options are to write a custom daemon or to use sesutil by hand. Instead of forcing all of us to write our own custom daemons, why not train zfsd to do it? My proposal is to add boolean GEOM attributes for "fault" and "locate". A userspace program would be able to look up their values for any geom with DIOCGATTR. Setting them would require a new ioctl (DIOCSATTR?). The disk class would issue a ENCIOC_SETELMSTAT to actually change the LEDs whenever this attribute changes. GEOM transforms such as geli would simply pass the attribute through to lower layers. Many-to-one transforms like gmultipath would pass the attribute through to all lower layers. zfsd could then set all vdevs' fault attributes when it starts up, and adjust individual disk's as appropriate on an event-driven basis. Questions: * Are there any obvious flaws in this plan, any reasons why GEOM attributes can't be used this way? * For one-to-many transforms like gpart the correct behavior is less clear: what if a disk has two partitions in two different pools, and one of them is healthy but the other isn't? * Besides ZFS, are there any other systems that could take advantage? * SATA enclosures uses SGPIO instead of SES. SGPIO is too limited, IMHO, to be of almost any use at all. I suggest not even trying to make it work with this scheme. From nobody Wed Feb 8 06:56:13 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PBW4W5CVfz3kpM9 for ; Wed, 8 Feb 2023 06:56:27 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PBW4W2zFmz3LGP; Wed, 8 Feb 2023 06:56:27 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102c.google.com with SMTP id f16-20020a17090a9b1000b0023058bbd7b2so1378300pjp.0; Tue, 07 Feb 2023 22:56:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:cc:date:in-reply-to:from:subject :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=3ySO/Tt+r/DxU6TkNC5vrPK42Rb4Cuzd+rC8poPqzeo=; b=IS3t+7WuokPmGm8b0V/gjy7oaFbF91/s6Lh6mIJMHAYkx0GchkzJ/cbNL6xm9Nmms9 8NXxC1awJXM6+Mfg1isZo7Lb5/fiJnMwkhymGmOpbNq1zFZMxloKT8UPsD+pnsiD4Sqg 4LWRX7+LPkmlHPtMMQ3eoGXgAlZ5a33gY2EUUd/j89rTwcDII6lAO/7OXV4+b5bZsG5o OJi5vFhyY5v/KfiCMXGc6xPb5Z4gEYp4jgJ1jOkywPnnSN+60ifbTXg3XglHAEHSBgpe 672mAhA6gsw+p/5DdsdQPb1YT6VfR1Wq/CfHycKUyX0pGQ5jgKoFgPreTV+mXnA6xmBS ZEcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:cc:date:in-reply-to:from:subject :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3ySO/Tt+r/DxU6TkNC5vrPK42Rb4Cuzd+rC8poPqzeo=; b=s2H9o8zteFL3DVgrr8N/A5dLgT9ufhEEX+CEWbi+xS3m5sozkxzuddIOUyz9od8Z88 NEL15BhLbWXKG7B0/jRtIsOt9CFqdFnqS7t88KoftxmJQASeWBvKR1yMOWndt/DDfNU+ ARalT+oC7cxgZrabTpZwE+woP6gHMZ4j+5OrcIBzKinkWf/4UVLflqATu8FfORpwgCSW iLWBtF2IKDR5fm1QlDWpCRxJJa8vtrNT1BcvQHYzzqYY9n3oU8OUx8ROowm85h2NREqI pbfM3uAyXidBqOblTe6J4RRVU5VUFGhq7hyIzzuv1fA2UKHWEPQlBPGD2m6g8u2pSN3f 59hA== X-Gm-Message-State: AO0yUKXmx7ZsX+MN4dM5ywc6AwBgaHYQI6mdQkOgcFH5LDjwktM93e5V 31Ow5ltuzcZf3OU4g5RFXKXitESPZ3JCYg== X-Google-Smtp-Source: AK7set+9+CdTxjXe544B1CRuk/eG+6n15t7IarRF7lUN+lUbe3XnV77KRlopCOGDvNroigPa+NYY4Q== X-Received: by 2002:a17:902:ec89:b0:196:7bfb:f0d1 with SMTP id x9-20020a170902ec8900b001967bfbf0d1mr5938917plg.34.1675839385314; Tue, 07 Feb 2023 22:56:25 -0800 (PST) Received: from smtpclient.apple (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id b4-20020a170902d30400b00196077ba463sm10031034plc.123.2023.02.07.22.56.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 07 Feb 2023 22:56:24 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_DB28F257-4E7D-401D-BDAE-ABD8D26EDCC6"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: RFC: GEOM and hard disk LEDs From: Enji Cooper In-Reply-To: Date: Tue, 7 Feb 2023 22:56:13 -0800 Cc: FreeBSD Hackers Message-Id: <0155CD68-849E-40D6-95A5-6AAD5E229A57@gmail.com> References: To: Alan Somers X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Rspamd-Queue-Id: 4PBW4W2zFmz3LGP X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_DB28F257-4E7D-401D-BDAE-ABD8D26EDCC6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Feb 7, 2023, at 3:30 PM, Alan Somers wrote: >=20 > Most modern SES backplanes have two LEDs per hard disk. There's a > "fault" LED and a "locate" LED. You can control either one with > sesutil(8) or, with a little more work, sg_ses from > sysutils/sg3_utils. They're very handy for tasks like replacing a > failed disk, especially in large enclosures. However, there isn't any > way to automatically control them. It would be very convenient if, > for example, zfsd(8) could do it. Basically, it would just set the > fault LED for any disk that has been kicked out of a ZFS pool, and > clear it for any disk that is healthy or is being resilvered. But > zfsd does not do that. Instead, users' only options are to write a > custom daemon or to use sesutil by hand. Instead of forcing all of us > to write our own custom daemons, why not train zfsd to do it? >=20 > My proposal is to add boolean GEOM attributes for "fault" and > "locate". A userspace program would be able to look up their values > for any geom with DIOCGATTR. Setting them would require a new ioctl > (DIOCSATTR?). The disk class would issue a ENCIOC_SETELMSTAT to > actually change the LEDs whenever this attribute changes. GEOM > transforms such as geli would simply pass the attribute through to > lower layers. Many-to-one transforms like gmultipath would pass the > attribute through to all lower layers. zfsd could then set all vdevs' > fault attributes when it starts up, and adjust individual disk's as > appropriate on an event-driven basis. >=20 > Questions: >=20 > * Are there any obvious flaws in this plan, any reasons why GEOM > attributes can't be used this way? >=20 > * For one-to-many transforms like gpart the correct behavior is less > clear: what if a disk has two partitions in two different pools, and > one of them is healthy but the other isn't? >=20 > * Besides ZFS, are there any other systems that could take advantage? >=20 > * SATA enclosures uses SGPIO instead of SES. SGPIO is too limited, > IMHO, to be of almost any use at all. I suggest not even trying to > make it work with this scheme. Out of curiosity, is there a way that a client of zfsd could = instead reach out to zfsd via a pipe, etc, when an event triggered by = devd occurs? Just trying to think of a way that would result in a cleaner = architecture than having zfsd doing a busy-loop of some kind waiting for = an event to happen. Thanks, -Enji --Apple-Mail=_DB28F257-4E7D-401D-BDAE-ABD8D26EDCC6 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEtvtxN6kOllEF3nmX5JFNMZeDGN4FAmPjR40ACgkQ5JFNMZeD GN6Ftg//cFCiDTr0UrqU7yPRSHNICezO6VjKHyPDBFjroIkmREmfVT3dDMhn1K3X y7TmsHLN6B9QrP9hr7HkCzHcAAME7Y7KTmL4Fva1uxUMwq+JNrGu1jERJKTLpoCB aLjTgWY3An7bK0Mg5qKTNTDpOyEIbp2seWhKzpyE6/enRaXzSFtm48TLoUAK6K4f pjMHXQxtL23+qInbkM9w1HDvK+eUU3v8u1LScVDloUQJp1rk2ySVJLfHu1wzdmua as/2mkX+JF48B9LKitX2vJGCcvPn6b9TucZEcBD9lXcMT50fpnk5KX0ts+HKRzO4 LfpoFP/6V1bOwrMlw+pged7UYpi1pmTmSbrOPgyOe3D/UH57uQH8l6mNKlWqjdC/ exoM5/7RNOAiQaTJsuewM4gGAmrixzTG6RPpNGsMHWioE8UWI8zQqRNlP9TB2+cF Vv3fQXad+MgsdKVsmKfkH6bDJTM0fACDQFSimmze7L3kNC105Pph/ZGcWDWQe7OW syrnHIcqDPuP2ElDCgRsQE+6MGlQEsoBokmQz8lkdjzlLFsn3WfisdMzLb7wSniy KHs2q2KOTamuHh0hoGd46IJUzW0b2yffLikONRg7Ao0opi1LSPFeRJfiy979Qd4s HI8ylwdc/y53N+MK8EWqhhvmvGl37jEwunAQR8MVtWXkgkszw7Q= =rYWb -----END PGP SIGNATURE----- --Apple-Mail=_DB28F257-4E7D-401D-BDAE-ABD8D26EDCC6-- From nobody Wed Feb 8 14:27:53 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PBj5f22JJz3nTWZ for ; Wed, 8 Feb 2023 14:28:06 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vk1-f177.google.com (mail-vk1-f177.google.com [209.85.221.177]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PBj5f0C1Pz4NCx for ; Wed, 8 Feb 2023 14:28:06 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-vk1-f177.google.com with SMTP id r28so2970198vka.4 for ; Wed, 08 Feb 2023 06:28:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=98LO4QJZRi9hCRKKJPa7fdR9ukzd1IVP2/Gy/Alv+W8=; b=O3LB9qvl0eiHtzhVp0FUCpJUvWfmDdJ25+IYoNb7BRwmtuTmRZDxHCZTMhN7Zv6JFF mL/meWCdAWkYSy+8VJknaLF/OOb43/Xlu4yn+2lykq/pvC1jpIZpiCTqzruFAEqqgkcu G8ZpGAPw846se2OY2bbG7k/waLVLnm0q8HFsM8Tx8GzJSBOmTa31fM7ebl8g705PzVP1 80KMoJ5Rg58ujYVInJb+h462k2q3AFJUZzfWQbCSOy4MAXvuPU/mxwVM028XTvPXYG5e zO3k2+p6Z6Ekz1eTnVl01bFeUwcVlxLHhyia5jPISIXI266/PcarlMvz6WMLPUskAHsq z1zQ== X-Gm-Message-State: AO0yUKWnWqRpIg1cOL0smn5OxZypoHkd08SKH9EllJByYTQ8kRGT52Pu /GqBVrpzq2VHaKN4EwNDyFGunGbqxFdJdjXSyTU= X-Google-Smtp-Source: AK7set+dEGMBHoMdzJ1o+Yc14ic1KgGL5wNF1DyUiB0Xi0xRKMNBAZcLR2Y7j6qkwJO/CLuaH0Hp27dP0F38TlkzywQ= X-Received: by 2002:a1f:274d:0:b0:3e8:791d:5ee with SMTP id n74-20020a1f274d000000b003e8791d05eemr1775225vkn.14.1675866484961; Wed, 08 Feb 2023 06:28:04 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <0155CD68-849E-40D6-95A5-6AAD5E229A57@gmail.com> In-Reply-To: <0155CD68-849E-40D6-95A5-6AAD5E229A57@gmail.com> From: Alan Somers Date: Wed, 8 Feb 2023 07:27:53 -0700 Message-ID: Subject: Re: RFC: GEOM and hard disk LEDs To: Enji Cooper Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4PBj5f0C1Pz4NCx X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Tue, Feb 7, 2023 at 11:56 PM Enji Cooper wrote: > > > > On Feb 7, 2023, at 3:30 PM, Alan Somers wrote: > > > > Most modern SES backplanes have two LEDs per hard disk. There's a > > "fault" LED and a "locate" LED. You can control either one with > > sesutil(8) or, with a little more work, sg_ses from > > sysutils/sg3_utils. They're very handy for tasks like replacing a > > failed disk, especially in large enclosures. However, there isn't any > > way to automatically control them. It would be very convenient if, > > for example, zfsd(8) could do it. Basically, it would just set the > > fault LED for any disk that has been kicked out of a ZFS pool, and > > clear it for any disk that is healthy or is being resilvered. But > > zfsd does not do that. Instead, users' only options are to write a > > custom daemon or to use sesutil by hand. Instead of forcing all of us > > to write our own custom daemons, why not train zfsd to do it? > > > > My proposal is to add boolean GEOM attributes for "fault" and > > "locate". A userspace program would be able to look up their values > > for any geom with DIOCGATTR. Setting them would require a new ioctl > > (DIOCSATTR?). The disk class would issue a ENCIOC_SETELMSTAT to > > actually change the LEDs whenever this attribute changes. GEOM > > transforms such as geli would simply pass the attribute through to > > lower layers. Many-to-one transforms like gmultipath would pass the > > attribute through to all lower layers. zfsd could then set all vdevs' > > fault attributes when it starts up, and adjust individual disk's as > > appropriate on an event-driven basis. > > > > Questions: > > > > * Are there any obvious flaws in this plan, any reasons why GEOM > > attributes can't be used this way? > > > > * For one-to-many transforms like gpart the correct behavior is less > > clear: what if a disk has two partitions in two different pools, and > > one of them is healthy but the other isn't? > > > > * Besides ZFS, are there any other systems that could take advantage? > > > > * SATA enclosures uses SGPIO instead of SES. SGPIO is too limited, > > IMHO, to be of almost any use at all. I suggest not even trying to > > make it work with this scheme. > > Out of curiosity, is there a way that a client of zfsd could instead reach out to zfsd via a pipe, etc, when an event triggered by devd occurs? > Just trying to think of a way that would result in a cleaner architecture than having zfsd doing a busy-loop of some kind waiting for an event to happen. > Thanks, > -Enji What do you mean a "client of zfsd"? The current architecture is that zfsd blocks waiting for events from devd. It reads from /var/run/devd.seqpacket.pipe. From nobody Thu Feb 9 12:08:49 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCFyY0Cztz3nlj7 for ; Thu, 9 Feb 2023 12:08:53 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCFyX6s1qz3Pbd for ; Thu, 9 Feb 2023 12:08:52 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675944533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=AEccue/VQFZACRhZNQQ7J9gs9Ha9yuWA3BRuyG3lVoA=; b=HUajpzCOUgWHNVe0cWvi1B+ncgjTEDbMCGf7OaEYgUP7rmZ7iY7rFDySIy3DMgJP1vxjr0 1oNLY/g1beF+K1daT5h0DPt79/XSFP2/iwuWTMGMEAQILBwmjhYPWXLrqOQMuImcUlCJx6 NoUOW0NOpawQWVGgHcqh5tx9IB+3QjwAtS0s+YBRJsCpGaijnJrUUqL2zP6c9D5HjozAbm xVLG6N+Yv7KUq8RlE7r1DmXyev9oxs0DFcaeKcOjqkzChD2sv9GqxHbky8KXbycOw8zNI6 br7D65O8YXfoQtzN1ABHbeo8EgF0Y08VhTcj1vbZARKfOz6wnGr467m4RV3Zeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675944533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=AEccue/VQFZACRhZNQQ7J9gs9Ha9yuWA3BRuyG3lVoA=; b=jXWhCDLJ/ngfFqGn2tPOm3c3JAlrGe6U3sT9IXfp5BXBD1y2pe4lSAGQbRRr3U6+OLobOq peuZsznW9jzo1rB91mYYVZeLYSJ6I7ilzqrIpLg5rmhdw6StsfMvPc8fP1hMcLJzhn9s0W MDiK2pVn+HzqYeWZZOTxIC7cCSvL/7jMe2ZB4ZAKoLhVaVYHwZUhQ/44IiWlpdj6MuyHC2 o57lpOvYQieKvI0mvsdFwtRmCatYSBlMMH12bGc7iZqmtcrGbrQVSoDZOU9HMdBzNEpFjp 3mZzgxBWT1dvymL9+uRXMivn1ilKYS+z68+MeFNMR1yRMjO+iyr4WJBk/tTipg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675944533; a=rsa-sha256; cv=none; b=YuA/2zslNb3And74q5mtnDfQy2RLl8OeUF/3OFtHOtvWvY9IV/yn7rcOoVOWIWSuHw9/I9 NG7Dk+iwUTFWv+pC/+jblznzPOGZtlZwproIi+fqGrglryuVMkKWzVsPWiSJeGNOFvnrbE bGWQHWhyKffGHJvOFDlLPclvjINbdfz5doVe9o20t5iNQAzEJ36dgmf9+2UUwxawnEQ/ym GPdBXm5caxj+XnHx8+sC+7xhhXUvIO/GblTLUnbClrpU+KkLujh4zTCJweUK6dw1mDGWWw Jpal3UT0EYSe+l7Nt3A6l81x7BEzagh1+3I84r5/JlwoKOSQsUk+s/1ymQ/fgw== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PCFyX5lybzN5K for ; Thu, 9 Feb 2023 12:08:52 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from [192.168.1.202] (host81-158-36-31.range81-158.btcentralplus.com [81.158.36.31]) by smtp.theravensnest.org (Postfix) with ESMTPSA id CF2251B5B for ; Thu, 9 Feb 2023 12:08:51 +0000 (GMT) Message-ID: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> Date: Thu, 9 Feb 2023 12:08:49 +0000 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Content-Language: en-GB To: freebsd-hackers From: David Chisnall Subject: CFT: snmalloc as libc malloc Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N Hi, For the few yearsI've been running locally with snmalloc as the malloc in libc. Eventually I'd like to propose this for upstreaming but it needs some wider testing first. For those unfamiliar with snmalloc (https://github.com/microsoft/snmalloc), it is an allocator (or, rather, a toolkit for building allocators) from my team at Microsoft Research designed for both performance and security. A few highlights: - Snmalloc uses a message-passing design, which makes allocating on one thread and freeing on another cheap. - Very fast allocation performance - Randomisation of relative locations of allocations - Most metadata is stored out-of-band - In-band metadata uses some lighweight encryption to protect against corruption. - Support for CHERI. In the (limited!) testing that I've done, it outperforms jemalloc and results in a smaller libc binary. I've also previously managed to use it in the kernel, though that code hasn't been tested in a while (last used with FreeBSD 11): https://github.com/microsoft/snmalloc/blob/main/src/snmalloc/pal/pal_freebsd_kernel.h It is also used in the Verona process sandboxing work, which makes it easy to isolate a library in a capsicum Sandbox: https://github.com/microsoft/verona/tree/master/experiments/process_sandbox We test on FreeBSD in CI upstream and the code is actively maintained. We have implemented compatibility wrappers for all of the jemalloc non-standard APIs that FreeBSD's libc exposes. In particular, snmalloc is designed to make it very cheap to find the start and end of an allocation, given a heap pointer. This means that we can insert bounds checks in critical libc functions to prevent heap overflow. This is done in the branch for memcpy, which some investigation of a corpus of security vulnerabilities showed was the root cause of about 10% of arbitrary-code-execution vulnerabilities. The bounds checks are controlled via an environment variable LIBC_BOUNDS_CHECKS. Setting this to 0 disables checks, to 1 checks on destination arguments, and to 2 checks sources and destinations. An ifunc resolver selects the correct memcpy implementation at load time. I did have a version that checked a bunch of other libc functions (e.g. sprintf, puts) but it was quite hacky (and the way the ifunc resolves was implemented broke tcl). The current branch puts two things behind the MALLOC_PRODUCTION toggle: - The additional security checks that detect corruption of malloc state. - Pretty-printing errors. We are currently separating the former into separate knobs upstream, some subset should probably be turned on by default in production. The latter has less of a performance impact than it had and will probably be on for all configurations at some point once we've refactored slightly to ensure the compiler can tail call the failure function (which moves it entirely off the fast path). With this enabled, you get errors that look like this: Fatal Error! memcpy with source out of bounds of heap allocation: range [0x14823c02440, 0x14823c0246a) allocation [0x14823c02440, 0x14823c02450) range goes beyond allocation by 0x1a bytes Abort trap (core dumped) Without it, you just get an illegal instruction trap. There are a few limitations in the current branch: - The memcpy integration is broken on non-amd64 platforms (patches welcome from people who can test these!). - Only memcpy (not, for example, memmove) has bounds checks. - The memcpy in rtld is naive, which may impact performance. - MALLOC_PRODUCTION conflates too many things The branch is here: https://github.com/davidchisnall/freebsd-src/tree/snmalloc2 It adds snmalloc as a submodule in contrib. FreeBSD is allergic to submodules, so upstreaming will need to replace this with something more complicated. You should be able to cherry-pick the top commit on any vaguely-recent -CURRENT. You should also be able to build the libc from this branch against the version that you're running and try it with LD_LIBRARY_PATH. I'd love to hear feedback on: - Performance, especially workloads where snmalloc does badly. - RSS usage (again, especially workloads where snmalloc does badly). - Anything that breaks. David From nobody Thu Feb 9 13:12:35 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCHN91y8Xz3p8h6 for ; Thu, 9 Feb 2023 13:12:41 +0000 (UTC) (envelope-from paulf2718@gmail.com) Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCHN82WgZz3pFy for ; Thu, 9 Feb 2023 13:12:40 +0000 (UTC) (envelope-from paulf2718@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=E+OXvxvX; spf=pass (mx1.freebsd.org: domain of paulf2718@gmail.com designates 2a00:1450:4864:20::329 as permitted sender) smtp.mailfrom=paulf2718@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-wm1-x329.google.com with SMTP id z13so1404208wmp.2 for ; Thu, 09 Feb 2023 05:12:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to:subject :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=6uM1+Af3tGki7eINwwbIV/To6O+Qs7w3+r/lC2keEhY=; b=E+OXvxvXBXnqyGmMa6C1/eRV7hJTtGz1a0GymTxaXwBPyHUP9xOL/LmaY5AxRaXF3A 2mBm/qx+bzEnFLO08jUnodgHaBQxAUw0wbjAUy7aba1uLEce5IeaW64EvZhO6P+G1XGW NvfhB8X9Cdgr894AURlpTkTc29WEOerYNbWnuioZNueV/qqkgTmgJttQFGZCdqf5gSnw 6RIqIov23/8rGm6eunmRlroCd4Koo2V7MnpLFIArrhDJfdBsTpCu6kAO0WitE98pb9bo Sl/hhuWjDkjZo+5/2LFSLI7412ObZPU16Z3gXPK1Elx/zIfiaxwtSKwcBwNmUxZinjaO vaHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=6uM1+Af3tGki7eINwwbIV/To6O+Qs7w3+r/lC2keEhY=; b=dR+ATk8FfrNy9T32U1/AjHBrD+QZF22pbXvSGeTMfkwAYzGiJnIKGZP/eJ3RAW91ED +A3OEst3FKCRg+/87s8mug+cy3Gwn2gvjg33xfTTPiA9Y/WYllh6bVioj9K3jBJriuf/ awqEm/ajvQCZXVeHpF5MjSvu9BtOMrJRX/WbpGaQiYWicINLRZ5BLGd9bUPT1nNcIQWu 2Rds9tvNN6YwiEYtl1BDnUWUqR0uBFNHK5FeP5Tcgtl50HjpwwRhNYmqKp4OrTp6dbr5 7hHpejR8GJBLG9uppKOzoBTYEOdJrYzesFsplPKEBTgZeKlB2YSkwrXGSkhhOmmhSCCb VLfg== X-Gm-Message-State: AO0yUKW0IAvp1ALPWwCOcatpvrZPM1EjnSBNibdpLezJxS1KGPGBmKq6 haTNfxnQtfpvc6RzGDaotbjBcUq486k83w== X-Google-Smtp-Source: AK7set+AUmi55nxssqVy8ZW7AlDsEh8UqpFsIQnJErpG50B45kUhrgMdwnwCpFibrUGkIDPIadf2tA== X-Received: by 2002:a05:600c:a686:b0:3e0:fad:5fa8 with SMTP id ip6-20020a05600ca68600b003e00fad5fa8mr9925388wmb.33.1675948358080; Thu, 09 Feb 2023 05:12:38 -0800 (PST) Received: from [137.202.253.23] (nat-ies.mentorg.com. [192.94.31.2]) by smtp.gmail.com with ESMTPSA id z6-20020a1c4c06000000b003d1d5a83b2esm4747121wmf.35.2023.02.09.05.12.37 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 09 Feb 2023 05:12:37 -0800 (PST) Message-ID: <3bacad17-ff6b-8bd3-0559-a70274dcd632@gmail.com> Date: Thu, 9 Feb 2023 14:12:35 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: CFT: snmalloc as libc malloc To: freebsd-hackers@freebsd.org References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> From: "Floyd, Paul" In-Reply-To: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.66 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.963]; NEURAL_HAM_MEDIUM(-0.70)[-0.699]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::329:from]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PCHN82WgZz3pFy X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 09/02/2023 13:08, David Chisnall wrote: > Hi, > > For the few yearsI've been running locally with snmalloc as the malloc > in libc.  Eventually I'd like to propose this for upstreaming but it > needs some wider testing first. > Hi Another allocator to add to my list. I'll give it a go some time, but I have a lot of things on my plate at the moment. I had a quick look. Do you support reallocf? Looking to the fairly short term future, I couldn't see anything for the upcoming C23 sized/aligned free functions. It may be premature as the standard isn't out yet. Recently I did some work on adding a warning for realloc of size 0 to Valgrind (not finished code review yet). I see that you 'free and return NULL' (in the same camp as glibc and ptmalloc). However, jemalloc is in the other camp "maybe free and return a pointer to something" (along with Solaris and musl). That's not really an issue since realloc of 0 is "implementation defined". However, it is slated to become UB in C23. A+ Paul From nobody Thu Feb 9 15:31:57 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCLSv5wh6z3pgLL for ; Thu, 9 Feb 2023 15:31:59 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCLSv5Kb9z45RX; Thu, 9 Feb 2023 15:31:59 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675956719; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mEgu3Zfc/CAk7PR/IJFTyaPaYnnV73pQKnoS6tJaDjA=; b=BI0gtlo898gBEDhopEP4BROFz5fY2DltNx/yd76fCBfARYfpSrAWrfp1P+CTA5jex0wJEa PSJiWJeK7nH6hsQdkvedG9PV5uxM6GCRg2SF6y8I565229EXh98BSRCZbIa+wFbG1MYstO Rx6k4ZChZbJB2EXUnYOomgGtJHzltV223B9e2qQt7y8u5WRXTY6+KjVfb2KUN5bASlyE8j 27GfRPZZEgAffHqOEvLv4TP+hJzC8SByvz6JbTNeeHwKBx+7DvKKjPU4aEJYX7jN0ZJF/r AIHZWXA/w/wJ2gjezNq3cUTwMn9DCT2lDsu8rat5JrWSKE5GNxWEbKdzbREyMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675956719; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mEgu3Zfc/CAk7PR/IJFTyaPaYnnV73pQKnoS6tJaDjA=; b=nQ+EgW7O6LhCFysB/zGQqrTeiYtHi5Nt13oqRNSFLlAy0gwFml/P8xqpC9HAeZuuSfIFby QAaEZV+Bz+qchcpgnR4HhG5W7xuCJxpAhB5zZ5v9Y4cdZtB5xekad1eUAdMurMOxFNdihb a6OcEIq3s99xmiUqBuZ6zwVunxhMbm5lACGTg2/gAoW4qCD1y5GBD7VqDsBi1LxP/lbYLg I+bjMlYCXMJZWk36IiUznHMpk67TQOkTWfSe0YfNPQUXnYsNKpTHbXi1WD1ai5ql7/8Pvv kOjgAkSBJdhyL2Ub/ffu5WYRRqUpm4olPE5yS5z9QnliTFFlcU4SBoBgCkBdcw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675956719; a=rsa-sha256; cv=none; b=yYzWRaBgsP5g9b+LfGMNRgtBb97qFZ42HqGrWBjmQndzkRfLv3ZQG0lmvseGpatow63nZr dqMXDjjiVnNxoBCy6l+G/SFROJoo0FcgWDtQSUqPQHXSOLSfC6fqJRQf7dR9NTg7sAE7PS j/0ErGF8IlthZ4jkbazKFQHkfdWsDdTLv2q18s6r058j7kgbeVW1INApLXCYfG8s3gY4rN qqbJfGc8lNurCj2rhUIjbml1y+TTAPkq7aglVmJ8RtBRpfB9E2rDuT8OQUJTNdu+9tMQF6 ySFdyO9KznGyrJ6bre8EdeqZd+fmR7tgtnUVF5YN/TLYfCs9vu+adjGy0kxeng== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PCLSv4NQgzQSl; Thu, 9 Feb 2023 15:31:59 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host81-158-36-31.range81-158.btcentralplus.com [81.158.36.31]) by smtp.theravensnest.org (Postfix) with ESMTPSA id EEECB1B5E; Thu, 9 Feb 2023 15:31:58 +0000 (GMT) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: CFT: snmalloc as libc malloc From: David Chisnall In-Reply-To: <3bacad17-ff6b-8bd3-0559-a70274dcd632@gmail.com> Date: Thu, 9 Feb 2023 15:31:57 +0000 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <2429423C-C66C-4466-AE47-AE72B67C41E9@FreeBSD.org> References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> <3bacad17-ff6b-8bd3-0559-a70274dcd632@gmail.com> To: "Floyd, Paul" X-Mailer: Apple Mail (2.3654.120.0.1.13) X-ThisMailContainsUnwantedMimeParts: N On 9 Feb 2023, at 13:12, Floyd, Paul wrote: >=20 > Another allocator to add to my list. I'll give it a go some time, but = I have a lot of things on my plate at the moment. >=20 > I had a quick look. Do you support reallocf? Not in snmalloc itself. FreeBSD libc supports this as a wrapper around = realloc: = https://github.com/freebsd/freebsd-src/blob/6332ef8941999b0c074d1ece0e1e10= 8447c70b98/lib/libc/stdlib/reallocf.c#L34 So libc using snmalloc does support it. > Looking to the fairly short term future, I couldn't see anything for = the upcoming C23 sized/aligned free functions. It may be premature as = the standard isn't out yet. Most of those can be implemented using the same underlying APIs as = posix_memalign, we can add them once they=E2=80=99re standardised. The = libc++ implementations of the C++ equivalents all simply wrap = posix_memalign. > Recently I did some work on adding a warning for realloc of size 0 to = Valgrind (not finished code review yet). I see that you 'free and return = NULL' (in the same camp as glibc and ptmalloc). However, jemalloc is in = the other camp "maybe free and return a pointer to something" (along = with Solaris and musl). That's not really an issue since realloc of 0 is = "implementation defined". However, it is slated to become UB in C23. We used to return NULL on malloc(0), and subsequently also on = realloc(0). We changed it because sort and (I think) pkg both call = realloc via a wrapper that checks for a NULL return and calls abort, yet = also pass 0 as an argument to realloc. This meant that I was unable to = buildworld after this, which was quite exciting (especially since I = forgot to create a boot environment before that update) - most other = things worked, but some key bits of FreeBSD didn=E2=80=99t. Our malloc code path could be shortened slightly if we could rely on = code accepting behaviour that conforms to the standard from malloc, = rather than the behaviour that glibc=E2=80=99s malloc happens to = provide. I was quite surprised to see this from code in the FreeBSD = base system, since none of the third-party code that I=E2=80=99ve used = with snmalloc had this behaviour. David From nobody Thu Feb 9 15:37:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCLcK14Lgz3pgxY for ; Thu, 9 Feb 2023 15:38:25 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay04.ispgateway.de (smtprelay04.ispgateway.de [80.67.18.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCLcJ0sgkz46xb for ; Thu, 9 Feb 2023 15:38:23 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of freebsd-listen@fabiankeil.de has no SPF policy when checking 80.67.18.16) smtp.mailfrom=freebsd-listen@fabiankeil.de; dmarc=none Received: from [91.20.76.172] (helo=fabiankeil.de) by smtprelay04.ispgateway.de with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pQ908-0003hv-Nq for freebsd-hackers@freebsd.org; Thu, 09 Feb 2023 16:38:20 +0100 Date: Thu, 9 Feb 2023 16:37:19 +0100 From: Fabian Keil To: freebsd-hackers@freebsd.org Subject: Re: CFT: snmalloc as libc malloc Message-ID: <20230209163719.55f13fa4@fabiankeil.de> In-Reply-To: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> Reply-To: freebsd-hackers@freebsd.org List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/8yqhgNut0ITKRUsLYKE=bxZ"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Df-Sender: Nzc1MDY3 X-Spamd-Result: default: False [0.71 / 15.00]; REPLYTO_EQ_TO_ADDR(5.00)[]; SIGNED_PGP(-2.00)[]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.99)[-0.993]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[80.67.18.16:from]; DMARC_NA(0.00)[fabiankeil.de]; HAS_REPLYTO(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; R_SPF_NA(0.00)[no SPF record]; RCVD_IN_DNSWL_NONE(0.00)[80.67.18.16:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:34011, ipnet:80.67.16.0/20, country:DE]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4PCLcJ0sgkz46xb X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N --Sig_/8yqhgNut0ITKRUsLYKE=bxZ Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable David Chisnall wrote on 2023-02-09 at 12:08:49: > For the few yearsI've been running locally with snmalloc as the malloc=20 > in libc. Eventually I'd like to propose this for upstreaming but it=20 > needs some wider testing first. Very interesting. > The branch is here: >=20 > https://github.com/davidchisnall/freebsd-src/tree/snmalloc2 >=20 > It adds snmalloc as a submodule in contrib. FreeBSD is allergic to=20 > submodules, so upstreaming will need to replace this with something more= =20 > complicated. You should be able to cherry-pick the top commit on any=20 > vaguely-recent -CURRENT. >=20 > You should also be able to build the libc from this branch against the=20 > version that you're running and try it with LD_LIBRARY_PATH. >=20 > I'd love to hear feedback on: >=20 > - Performance, especially workloads where snmalloc does badly. > - RSS usage (again, especially workloads where snmalloc does badly). > - Anything that breaks. Do you know how much work it would be to test with 13/stable instead of current? For a while now I have been collecting Privoxy TLS benchmarks [0] using ElectroBSD (which is currently based on FreeBSD stable/13) and various TLS libraries and would like to know if snmalloc affects the benchmark results. Fabian [0] --Sig_/8yqhgNut0ITKRUsLYKE=bxZ Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQTKUNd6H/m3+ByGULIFiohV/3dUnQUCY+UTLwAKCRAFiohV/3dU nd3WAKCLp68WpwxEea7qW3n3f4WGGuCnNQCgvsMHIKLNGpZ1v41Bs8BWCpjYiEg= =WZWS -----END PGP SIGNATURE----- --Sig_/8yqhgNut0ITKRUsLYKE=bxZ-- From nobody Thu Feb 9 19:15:24 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCRQk5MRJz3nQx3 for ; Thu, 9 Feb 2023 19:15:26 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x235.google.com (mail-oi1-x235.google.com [IPv6:2607:f8b0:4864:20::235]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCRQk3S3tz3Q9k; Thu, 9 Feb 2023 19:15:26 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x235.google.com with SMTP id cz14so2488277oib.12; Thu, 09 Feb 2023 11:15:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=IyYiYPsyFbF0//ohyHC9rKadKqzxR4dXIB6S6vQ7dMo=; b=HRY19yc2uGGjYMOeOqTvE4aSuEIixcAbkE6a0mf+FdDg80cyqR3u37Dra39A54tbLU cz3/Qt1BhiORodLQW378xMv/9SwN9nXyt33c9RNp+63Vh+20NsSwgq9zdinpk1aaDf/a L55xgwMt3vVlYux3mF8R0sD8SCr0H372JkX0k9M81CkPQ/1ki6IkI1HKkOJBh5hsQNx+ +rK0jMew2ghCOlqEUxJnghh4KhILMqoofp5njUjBDn7nVeslSVylAxDo4SoJWZJgJDnl M9Q/BkrdbdX7E9kZ0g1uZt/FxUwiGvE6j8sjCEGck+ISAVLuesN/LV3rj0iFHcCmdVF9 CSgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IyYiYPsyFbF0//ohyHC9rKadKqzxR4dXIB6S6vQ7dMo=; b=M0RX54Q9m8bSarcn9n4MeSijpDS90lWWU0TMkK+gkal9mn/L3cXwRq+J94CoVBD9ew SWJkphVSPl8gQGfh2VZ9YObZ4FqBcD47pKAg3fCNJXCBz6kjWyaNjYPbnB+b3QKv7bIE MjoKhPa6EdXO/PVqol2up9kmhcnu16xYewj0YX/d349/Fduk/qAkXWsuv8OgEmmRDBjf OXg9OKStP3+oJcF9K2SrzLczgpUWdQkGk/Q3qvzzVL07Tajl/9Vt5QU/uS9z3GCvNC+e OCQBEmM++kPNV/IVtETgBzQ9hBhDEwB8jl1FOzVAgHuMUChL6DcXJ5ScFt1FgyOXSzB0 Qy9Q== X-Gm-Message-State: AO0yUKURORUOyCfsn3AWPRwJFUXqyuCj82Fr3QVDDgn8AIyjhfl4WWeo 6FS8zXNlhoO7D9+wSgdnNXnLrk9V4LR+pcWbeppejUM4 X-Google-Smtp-Source: AK7set+QSIM31QvwIzu0QJKFT5fUSiks+6VNADWalr0XWjIyQDe1+VrvKDptg0OloOQpIfyO0F/Q7ROsEfJ5mdqP058= X-Received: by 2002:aca:3c05:0:b0:378:348b:9346 with SMTP id j5-20020aca3c05000000b00378348b9346mr476773oia.81.1675970125156; Thu, 09 Feb 2023 11:15:25 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:6c92:0:b0:4b3:d953:974c with HTTP; Thu, 9 Feb 2023 11:15:24 -0800 (PST) In-Reply-To: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> From: Mateusz Guzik Date: Thu, 9 Feb 2023 20:15:24 +0100 Message-ID: Subject: Re: CFT: snmalloc as libc malloc To: David Chisnall Cc: freebsd-hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4PCRQk3S3tz3Q9k X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 2/9/23, David Chisnall wrote: > Hi, > > For the few yearsI've been running locally with snmalloc as the malloc > in libc. Eventually I'd like to propose this for upstreaming but it > needs some wider testing first. > > For those unfamiliar with snmalloc > (https://github.com/microsoft/snmalloc), it is an allocator (or, rather, > a toolkit for building allocators) from my team at Microsoft Research > designed for both performance and security. A few highlights: > > - Snmalloc uses a message-passing design, which makes allocating on > one thread and freeing on another cheap. > - Very fast allocation performance > - Randomisation of relative locations of allocations > - Most metadata is stored out-of-band > - In-band metadata uses some lighweight encryption to protect against > corruption. > - Support for CHERI. > > In the (limited!) testing that I've done, it outperforms jemalloc and > results in a smaller libc binary. > > I've also previously managed to use it in the kernel, though that code > hasn't been tested in a while (last used with FreeBSD 11): > > https://github.com/microsoft/snmalloc/blob/main/src/snmalloc/pal/pal_freebsd_kernel.h > > It is also used in the Verona process sandboxing work, which makes it > easy to isolate a library in a capsicum Sandbox: > > https://github.com/microsoft/verona/tree/master/experiments/process_sandbox > > We test on FreeBSD in CI upstream and the code is actively maintained. > We have implemented compatibility wrappers for all of the jemalloc > non-standard APIs that FreeBSD's libc exposes. > > In particular, snmalloc is designed to make it very cheap to find the > start and end of an allocation, given a heap pointer. This means that > we can insert bounds checks in critical libc functions to prevent heap > overflow. This is done in the branch for memcpy, which some > investigation of a corpus of security vulnerabilities showed was the > root cause of about 10% of arbitrary-code-execution vulnerabilities. > > The bounds checks are controlled via an environment variable > LIBC_BOUNDS_CHECKS. Setting this to 0 disables checks, to 1 checks on > destination arguments, and to 2 checks sources and destinations. An > ifunc resolver selects the correct memcpy implementation at load time. > > I did have a version that checked a bunch of other libc functions (e.g. > sprintf, puts) but it was quite hacky (and the way the ifunc resolves > was implemented broke tcl). > > The current branch puts two things behind the MALLOC_PRODUCTION toggle: > > - The additional security checks that detect corruption of malloc state. > - Pretty-printing errors. > > We are currently separating the former into separate knobs upstream, > some subset should probably be turned on by default in production. The > latter has less of a performance impact than it had and will probably be > on for all configurations at some point once we've refactored slightly > to ensure the compiler can tail call the failure function (which moves > it entirely off the fast path). With this enabled, you get errors that > look like this: > > Fatal Error! > memcpy with source out of bounds of heap allocation: > range [0x14823c02440, 0x14823c0246a) > allocation [0x14823c02440, 0x14823c02450) > range goes beyond allocation by 0x1a bytes > > Abort trap (core dumped) > > Without it, you just get an illegal instruction trap. > > There are a few limitations in the current branch: > > - The memcpy integration is broken on non-amd64 platforms (patches > welcome from people who can test these!). > - Only memcpy (not, for example, memmove) has bounds checks. > - The memcpy in rtld is naive, which may impact performance. > - MALLOC_PRODUCTION conflates too many things > > The branch is here: > > https://github.com/davidchisnall/freebsd-src/tree/snmalloc2 > > It adds snmalloc as a submodule in contrib. FreeBSD is allergic to > submodules, so upstreaming will need to replace this with something more > complicated. You should be able to cherry-pick the top commit on any > vaguely-recent -CURRENT. > > You should also be able to build the libc from this branch against the > version that you're running and try it with LD_LIBRARY_PATH. > > I'd love to hear feedback on: > > - Performance, especially workloads where snmalloc does badly. > - RSS usage (again, especially workloads where snmalloc does badly). > - Anything that breaks. > it fails to build for me: /usr/src/lib/libc/stdlib/snmalloc/malloc.cc:35:10: fatal error: 'override/jemalloc_compat.cc' file not found #include "override/jemalloc_compat.cc" ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 error generated. --- malloc.o --- *** [malloc.o] Error code 1 make[4]: stopped in /usr/src/lib/libc /usr/src/lib/libc/stdlib/snmalloc/memcpy.cc:25:10: fatal error: 'global/memcpy.h' file not found #include ^~~~~~~~~~~~~~~~~ 1 error generated. --- memcpy.o --- *** [memcpy.o] Error code 1 this is a fresh world, top of snmalloc2 branch: commit a5c83c69817d03943b8be982dd815c7e263d1a83 Author: David Chisnall Date: Fri Jan 21 15:13:09 2022 +0000 Initial commit of snmalloc2 in libc. anyway, I wanted to say I find the memcpy thing incredibly suspicious. I found one article in https://github.com/microsoft/snmalloc/blob/main/docs/security/GuardedMemcpy.md which benches it and that made it even more suspicious. How did the benched memcpy look like inside? -- Mateusz Guzik From nobody Thu Feb 9 19:30:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCRmP4y83z3nT4b for ; Thu, 9 Feb 2023 19:30:45 +0000 (UTC) (envelope-from embhd@posteo.de) Received: from mout02.posteo.de (mout02.posteo.de [185.67.36.66]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCRmN4Wt1z3kFM for ; Thu, 9 Feb 2023 19:30:44 +0000 (UTC) (envelope-from embhd@posteo.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=posteo.de header.s=2017 header.b=f36UVO2t; spf=pass (mx1.freebsd.org: domain of embhd@posteo.de designates 185.67.36.66 as permitted sender) smtp.mailfrom=embhd@posteo.de; dmarc=pass (policy=none) header.from=posteo.de Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 00844240205 for ; Thu, 9 Feb 2023 20:30:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.de; s=2017; t=1675971042; bh=CqqFFDyL4+wnO/FvjJmaUSx45gNR7pxoOr7LPNpyeiA=; h=Date:From:To:Subject:From; b=f36UVO2tY86wCQ3iLHfJa/FeKHhj+gyv+iybCQowvHgdFMfDDa7VjfYoAP8HahqdD QGuc7Stm1Wm/iRc/43yS9A6qQ3ujb1pXYgSbGMPM/tIVqeHwhhwdcxerEzIdHdgthk jWH6qAm+tRakIC2B39/7rbLWlgXZwPnabeypsQyZ/ogpyWIiOt+gilE5PBOEz437fj JAxnZr8AEjMmuS1IcL4MG8vV4v4DiwrGTQ3KD/KKkOdqTImGqghcKkGC7tu1OFIgbY DEzYqZSw0PKBZ8nC2Ea0q0qH9Q4tOq0hX/V8fWXJSdZzjcWlMFn7F1r9Fw5K96Rzar szqN7174Xpojg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4PCRmK4K2mz9rxG for ; Thu, 9 Feb 2023 20:30:41 +0100 (CET) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_69eafa2e739677108a0d6025b4da0ca0" Date: Thu, 09 Feb 2023 19:30:41 +0000 From: Michael To: freebsd-hackers@freebsd.org Subject: nanoBSD: =?UTF-8?Q?cust=5Fpkgng=28=29=20fetches=20ports-mgmt/pkg?= =?UTF-8?Q?=20regardless=20of=20required=20package?= Message-ID: <81d8f972bb1738771716aa77f207bbce@posteo.de> X-Spamd-Result: default: False [-3.39 / 15.00]; SUBJ_EXCESS_QP(1.20)[]; DWL_DNSWL_LOW(-1.00)[posteo.de:dkim]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[posteo.de,none]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; R_DKIM_ALLOW(-0.20)[posteo.de:s=2017]; RCVD_IN_DNSWL_MED(-0.20)[185.67.36.66:from]; R_SPF_ALLOW(-0.20)[+ip4:185.67.36.0/23]; MIME_BASE64_TEXT(0.10)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain,text/x-patch]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; HAS_ATTACHMENT(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[posteo.de:+]; MIME_TRACE(0.00)[0:+,1:+,2:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:8495, ipnet:185.67.36.0/23, country:DE]; RWL_MAILSPIKE_POSSIBLE(0.00)[185.67.36.66:from]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4PCRmN4Wt1z3kFM X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --=_69eafa2e739677108a0d6025b4da0ca0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed Good evening, I noticed that in cust_pkng(), pkg(7) (/usr/sbin/pkg) fetches pkg(8) (ports-mgmt/pkg) from remote even though the function itself requires a binary package of pkg(8) for bootstrapping. This also seems to be the reason why _.cust.cust_pkng notes that "the most recent version of pkg-X.Y.Z is already installed" - first, pkg(7) fetches and builds pkg(8), which then tries to install pkg(8) once more. At the same time, I would guess that if the nanoBSD-image is for an entirely different plattform/architecture, the just bootstrapped pkg(8) would not be able to run during later operation, since I don't think it has been compiled for the later host system. My current solution is to change the command set in ${PKGCMD} to use the build system's pkg(8), which chroots itself to ${NANO_WORLDDIR}. This also makes the separate calls to chroot(8) by means of CR() and CR0() unnecessary. Since the latter does not seem to be used anywhere else and the former only once in clean_build(), they can be removed altogether if the remaining call to CR() in clean_build() is replaced with a direct call to chroot(8). I suppose this is also somewhat related to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260467, since the corresponding line in cust_pkgng() is `CR "${PKGCMD} add /_.p/${_NANO_PKG_PACKAGE}"`. With the solution above, pkg(7) would never be invoked in the first place and the bug wouldn't pop up, at least in the default function. Am I missing anything? Regards, Michael PS: I originally posted this (minus some re-wording) to freebsd-embedded about two weeks ago, but never received any response. --=_69eafa2e739677108a0d6025b4da0ca0 Content-Transfer-Encoding: base64 Content-Type: text/x-patch; name=defaults.patch Content-Disposition: attachment; filename=defaults.patch; size=2023 ZGlmZiAtLWdpdCBhL3Rvb2xzL3Rvb2xzL25hbm9ic2QvZGVmYXVsdHMuc2ggYi90b29scy90b29s cy9uYW5vYnNkL2RlZmF1bHRzLnNoCmluZGV4IDQ1ZDlmZTQ0YzY1MC4uZTVkNmNhY2RiMzQ0IDEw MDc1NQotLS0gYS90b29scy90b29scy9uYW5vYnNkL2RlZmF1bHRzLnNoCisrKyBiL3Rvb2xzL3Rv b2xzL25hbm9ic2QvZGVmYXVsdHMuc2gKQEAgLTI3NywxNiArMjc3LDYgQEAgdGd0X2RpcjJzeW1s aW5rICggKSAoCiAJZmkKICkKIAotIyBydW4gaW4gdGhlIHdvcmxkIGNocm9vdCwgZXJyb3JzIGZh dGFsCi1DUiAoICkgewotCWNocm9vdCAiJHtOQU5PX1dPUkxERElSfSIgL2Jpbi9zaCAtZXhjICIk KiIKLX0KLQotIyBydW4gaW4gdGhlIHdvcmxkIGNocm9vdCwgZXJyb3JzIG5vdCBmYXRhbAotQ1Iw ICggKSB7Ci0JY2hyb290ICIke05BTk9fV09STERESVJ9IiAvYmluL3NoIC1jICIkKiIgfHwgdHJ1 ZQotfQotCiBjbGVhbl9idWlsZCAoICkgKAogCXBwcmludCAyICJDbGVhbiBhbmQgY3JlYXRlIG9i amVjdCBkaXJlY3RvcnkgKCR7TUFLRU9CSkRJUlBSRUZJWH0pIgogCkBAIC03NDAsNyArNzMwLDcg QEAgY3VzdF9pbnN0YWxsX2ZpbGVzICggKSAoCiAJZmluZCAuIC1wcmludCB8IGdyZXAgLUV2ICcv KENWU3xcLnN2bnxcLmhnfFwuZ2l0KS8nIHwgY3BpbyAke0NQSU9fU1lNTElOS30gLUxkdW1wdiAk e05BTk9fV09STERESVJ9CiAKIAlpZiBbIC1uICIke05BTk9fQ1VTVF9GSUxFU19NVFJFRX0iIC1h IC1mICR7TkFOT19DVVNUX0ZJTEVTX01UUkVFfSBdOyB0aGVuCi0JCUNSICJtdHJlZSAtZWlVIC1w IC8iIDwke05BTk9fQ1VTVF9GSUxFU19NVFJFRX0KKwkJY2hyb290ICIke05BTk9fV09STERESVJ9 IiAvYmluL3NoIC1leGMgIm10cmVlIC1laVUgLXAgLyA8JHtOQU5PX0NVU1RfRklMRVNfTVRSRUV9 IgogCWZpCiApCiAKQEAgLTc1MCw3ICs3NDAsNyBAQCBjdXN0X2luc3RhbGxfZmlsZXMgKCApICgK IGN1c3RfcGtnbmcgKCApICgKIAlta2RpciAtcCAke05BTk9fV09STERESVJ9L3Vzci9sb2NhbC9l dGMKIAlsb2NhbCBQS0dfQ09ORj0iJHtOQU5PX1dPUkxERElSfS91c3IvbG9jYWwvZXRjL3BrZy5j b25mIgotCWxvY2FsIFBLR0NNRD0iZW52IEJBVENIPVlFUyBBU1NVTUVfQUxXQVlTX1lFUz1ZRVMg UEtHX0RCRElSPSR7TkFOT19QS0dfTUVUQV9CQVNFfS9wa2cgU0lHTkFUVVJFX1RZUEU9bm9uZSAv dXNyL3NiaW4vcGtnIgorCWxvY2FsIFBLR0NNRD0iZW52IEJBVENIPVlFUyBBU1NVTUVfQUxXQVlT X1lFUz1ZRVMgUEtHX0RCRElSPSR7TkFOT19QS0dfTUVUQV9CQVNFfS9wa2cgU0lHTkFUVVJFX1RZ UEU9bm9uZSBwa2cgLWMgJHtOQU5PX1dPUkxERElSfSIKIAogCSMgRW5zdXJlIHBrZy5jb25mIHBv aW50cyBwa2cgdG8gd2hlcmUgdGhlIHBhY2thZ2UgbWV0YSBkYXRhIGxpdmVzLgogCXRvdWNoICR7 UEtHX0NPTkZ9CkBAIC03ODMsNyArNzczLDcgQEAgY3VzdF9wa2duZyAoICkgKAogCXRyYXAgInVt b3VudCAke05BTk9fV09STERESVJ9L2RldjsgdW1vdW50ICR7TkFOT19XT1JMRERJUn0vXy5wIDsg cm0gLXhyZiAke05BTk9fV09STERESVJ9L18ucCIgMSAyIDE1IEVYSVQKIAogCSMgSW5zdGFsbCBw a2ctKiBwYWNrYWdlCi0JQ1IgIiR7UEtHQ01EfSBhZGQgL18ucC8ke19OQU5PX1BLR19QQUNLQUdF fSIKKwkke1BLR0NNRH0gYWRkIC9fLnAvJHtfTkFOT19QS0dfUEFDS0FHRX0KIAogCSgKIAkJIyBF eHBhbmQgYW55IGdsb2IgY2hhcmFjdGVycyBpbiBwYWNha2dlIGxpc3QKQEAgLTc5OCwxMSArNzg4 LDExIEBAIGN1c3RfcGtnbmcgKCApICgKIAogCQkjIEluc3RhbGwgcGFja2FnZXMKIAkJZm9yIF9Q S0cgaW4gJF9QS0dTOyBkbwotCQkJQ1IgIiR7UEtHQ01EfSBhZGQgL18ucC8ke19QS0d9IgorCQkJ JHtQS0dDTUR9IGFkZCAvXy5wLyR7X1BLR30KIAkJZG9uZQogCSkKIAotCUNSMCAiJHtQS0dDTUR9 IGluZm8iCisJJHtQS0dDTUR9IGluZm8gfHwgdHJ1ZQogCiAJdHJhcCAtIDEgMiAxNSBFWElUCiAJ dW1vdW50ICR7TkFOT19XT1JMRERJUn0vZGV2Cg== --=_69eafa2e739677108a0d6025b4da0ca0-- From nobody Thu Feb 9 19:36:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCRvR4CPtz3nTlP for ; Thu, 9 Feb 2023 19:36:51 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCRvR3gPqz3mBw; Thu, 9 Feb 2023 19:36:51 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675971411; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZMst3TDKpRdZkQ8z3KR3QzMYccqCX00YrRdDckSxE4Y=; b=AT4unnoDFGjpUoMNdirBBoWAXXfvXqPC4VGgWVFJ1BvZU5O6R45cW4/0nXQYpVIWWlyg+n YNfNoPBrlKma0QfHdOrlzdcl2Vy6/mmtPkx9th8mjzstAOoKb+qE9UcPZChvtFtHFBQLHf 1qD3tSJEOEZb2VvMDI2RtEi7U4ENWOCmvA/rHZACbSlodHOS34h04QlUVQxvHzLwRRHJji kn+WrPfyZEhwqfJYmaDMTPsEGgufiJOSE7KzOpw8rXMtDeOGvOJj3I1L99uBFZhy2MY72q HgSEJUV3Ba/rZsUk6epzS42BAp2uhBBi+DNLnCq9Ki3LQr8R0XeIhU0j7zhA2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675971411; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZMst3TDKpRdZkQ8z3KR3QzMYccqCX00YrRdDckSxE4Y=; b=j15biPAbB23orzbcc84aPZpCfFpNGbq6aICWu8BxiHqYR0lS4rnC6owEUfzV7zjeCjFQvW PEqX3xYO4G9EUAP2D/telOM1UDMCFHasb04pgQpVWPALiCmgoPIIThDpL3iONFW+tJILhl //K+AoXlnnj7RD9pVNuR8Ya0yy5K2FLIjNhhV4AyaxQIy8kdh2gJxruRFrp4ALcodfvmCh FMiwhT1Fh7mnE5kj4GTZgJcLo91129OwcXIQmioGdTKfbwOZfyba4A8xJC5skLyQryEJI+ Abz6BMRDkgua5vyDlZPQsmf5C3ggfJuJlWWvEBO7aoieUuzwRZNgh+EivDn/fw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675971411; a=rsa-sha256; cv=none; b=b7wTWpbzyMA3N+PJIOB32GUpobWMoEjLMjhT1u10H0Sh0Ri3+2YNXLixGdUJqbsHjg5OhQ Jen4N0Qa+/vk8AeQuCJTzcyfsgbuYGQubQDZGw4MZymZQr1OuH5u9HLn7OVImskqbC5Xyr eylYpb08P9SNArlzdqWKQAG1PhJf7OtrxaQqbW8fpRWcnMefyQvXUsoLmRxDflHtnh78aI A59vzTMnZQNcwcxkqrqRuqeqzvHysTYeLPLrUzJjWiD8EIEQnB9RbL69HVHW++s4A9C/pD YNh7CgGijwbJoVVy7ni/huDfC/uThQLxVA4zEJEr8lUFXSWKt18IvfzwHW2hVw== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PCRvR2b2NzV8J; Thu, 9 Feb 2023 19:36:51 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host81-158-36-31.range81-158.btcentralplus.com [81.158.36.31]) by smtp.theravensnest.org (Postfix) with ESMTPSA id AF3F21058A; Thu, 9 Feb 2023 19:36:49 +0000 (GMT) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: CFT: snmalloc as libc malloc From: David Chisnall In-Reply-To: Date: Thu, 9 Feb 2023 19:36:48 +0000 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> To: Mateusz Guzik X-Mailer: Apple Mail (2.3654.120.0.1.13) X-ThisMailContainsUnwantedMimeParts: N On 9 Feb 2023, at 19:15, Mateusz Guzik wrote: >=20 > it fails to build for me: >=20 > /usr/src/lib/libc/stdlib/snmalloc/malloc.cc:35:10: fatal error: > 'override/jemalloc_compat.cc' file not found > #include "override/jemalloc_compat.cc" > ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > 1 error generated. > --- malloc.o --- > *** [malloc.o] Error code 1 >=20 > make[4]: stopped in /usr/src/lib/libc > /usr/src/lib/libc/stdlib/snmalloc/memcpy.cc:25:10: fatal error: > 'global/memcpy.h' file not found > #include > ^~~~~~~~~~~~~~~~~ > 1 error generated. > --- memcpy.o --- > *** [memcpy.o] Error code 1 This looks as if you haven=E2=80=99t got the submodule? Is there = anything in contrib/snmalloc? > this is a fresh world, top of snmalloc2 branch: > commit a5c83c69817d03943b8be982dd815c7e263d1a83 > Author: David Chisnall > Date: Fri Jan 21 15:13:09 2022 +0000 >=20 > Initial commit of snmalloc2 in libc. >=20 > anyway, I wanted to say I find the memcpy thing incredibly suspicious. > I found one article in > = https://github.com/microsoft/snmalloc/blob/main/docs/security/GuardedMemcp= y.md > which benches it and that made it even more suspicious. How did the > benched memcpy look like inside? Perhaps you could share what you are suspicious about? I don=E2=80=99t = really know how to respond to something so vague. The document you = linked to has the benchmark that we used (though the graphs in it appear = to be based on an older version of the memcpy). The PR that added = PowerPC tuning has some additional graphs of measurements. If you compile the memcpy file, you can see the assembly. The C++ = provides a set of building blocks for producing efficient memcpy = implementations. The fastest on x86 is roughly: - A jump table of power for small sizes that do power-of-two-sized = small copies (double-word, word, half-word, and byte) to perform the = copy. - A vectorised copy for medium-sized copies using a loop of SSE copies. - rep movsb for large copies. The compiler does some quite complex layout for the jump table. David From nobody Thu Feb 9 19:45:51 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCS5x5VPVz3nW97 for ; Thu, 9 Feb 2023 19:45:57 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: from spindle.one-eyed-alien.net (spindle.one-eyed-alien.net [199.48.129.229]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCS5x3XXsz3p2s; Thu, 9 Feb 2023 19:45:57 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Authentication-Results: mx1.freebsd.org; none Received: by spindle.one-eyed-alien.net (Postfix, from userid 3001) id 3AFF63C0199; Thu, 9 Feb 2023 19:45:51 +0000 (UTC) Date: Thu, 9 Feb 2023 19:45:51 +0000 From: Brooks Davis To: David Chisnall Cc: Mateusz Guzik , freebsd-hackers Subject: Re: CFT: snmalloc as libc malloc Message-ID: References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: X-Rspamd-Queue-Id: 4PCS5x3XXsz3p2s X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36236, ipnet:199.48.128.0/22, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Feb 09, 2023 at 07:36:48PM +0000, David Chisnall wrote: > On 9 Feb 2023, at 19:15, Mateusz Guzik wrote: > >=20 > > it fails to build for me: > >=20 > > /usr/src/lib/libc/stdlib/snmalloc/malloc.cc:35:10: fatal error: > > 'override/jemalloc_compat.cc' file not found > > #include "override/jemalloc_compat.cc" > > ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > 1 error generated. > > --- malloc.o --- > > *** [malloc.o] Error code 1 > >=20 > > make[4]: stopped in /usr/src/lib/libc > > /usr/src/lib/libc/stdlib/snmalloc/memcpy.cc:25:10: fatal error: > > 'global/memcpy.h' file not found > > #include > > ^~~~~~~~~~~~~~~~~ > > 1 error generated. > > --- memcpy.o --- > > *** [memcpy.o] Error code 1 >=20 > This looks as if you haven???t got the submodule? Is there anything in c= ontrib/snmalloc? I'd suggest adding a Makefile check for contrib/snmalloc/LICENSE or the like with an .error with appropriate git submodule commands. -- Brooks From nobody Thu Feb 9 19:46:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCS6Z73QBz3nVwM for ; Thu, 9 Feb 2023 19:46:30 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCS6Z5NYVz3q68 for ; Thu, 9 Feb 2023 19:46:30 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62d.google.com with SMTP id ud5so9740244ejc.4 for ; Thu, 09 Feb 2023 11:46:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QN0nwEGlmDwHpF6jDQanZpOi8UQsgzItg62WPcp+nH4=; b=hxzaFLEmxeM+tzQAi0yh5nOoafmlW2l0p/TgfW6eFIsMR0miF/70du8Snb8SIGwuS1 UXPV3Hi9Q7OWbZRUAFLPjKraMhn3ZoRQn/GIWuAdklhu4BiLCI6KEYtZ+a+exHJ1Bcmi JUBoEcI8V+rRFb2TbmHBIoqyBpPL5wJiq2YzLGGgKiYY8D+4NuP76YIrImE3iKtb8ZZQ 1ICLQV46RWSFvHOsLJ9Y5qJOL/J3SZMgbFB03oyHs/EYwLp+LngbdJXMFEjbNDrsbvlJ 12Ca9vS5xK2tgehVEUcXcI2pE6fp0BQoyNrOb/o+WJpJ30IUFqh6OuwrzAtWG0xul0vg 20Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QN0nwEGlmDwHpF6jDQanZpOi8UQsgzItg62WPcp+nH4=; b=5+O7CgPJI0aQAnIb8WhU5LA1KSISodIcVVzCN93+j/uP2FFLGQm9OXlLoh7+KYxNBC pMo5k4pljp54jbWrv0zbxfe3UZLUZHn6XJubt6FmJdcc2+lluqG6g3zhTSJ859ahES0p v6SX9ErLlNlYrWXUAJWr/Rktrb4+T3DAiP+TnIcWwXw1X9SQlusO/G3xM1G88IXEZx61 sT61Yw+a4aYHbSkqC1VQoI+DSLKSGP8gItevOVIcaYVwVhnG9Ro18B5wDQ56nQGO2uIl Jno9c5ur+aBuLBda3wkRKReGu+ZaiScVZvqTSQrmsAoBxckVguCHnjK+hCJJcfmqWcPk 62Cg== X-Gm-Message-State: AO0yUKWAsCcr3Wy5ey6E8S6TFHcAFfwK2hZo/RlFPcekKfu2/aP0Z3Cl s4WOwm7zFcX/UJNHHv6FElTd3Hn5EBd0T74/g8uPyrqyTH+HJw== X-Google-Smtp-Source: AK7set9oo8XzGvxFONpBWhxJokMMVhqRSAI22MxvxKYPipltfxCwJ6xYkydhooL1BUv7HMuXtOrfroYHSgFZfu0qZzo= X-Received: by 2002:a17:906:2f17:b0:888:c14e:70b6 with SMTP id v23-20020a1709062f1700b00888c14e70b6mr2877113eji.306.1675971989063; Thu, 09 Feb 2023 11:46:29 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> In-Reply-To: From: Warner Losh Date: Thu, 9 Feb 2023 12:46:16 -0700 Message-ID: Subject: Re: CFT: snmalloc as libc malloc To: David Chisnall Cc: Mateusz Guzik , freebsd-hackers Content-Type: multipart/alternative; boundary="0000000000000fdb0105f449a32b" X-Rspamd-Queue-Id: 4PCS6Z5NYVz3q68 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000000fdb0105f449a32b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Feb 9, 2023, 12:37 PM David Chisnall wrote: > On 9 Feb 2023, at 19:15, Mateusz Guzik wrote: > > > > it fails to build for me: > > > > /usr/src/lib/libc/stdlib/snmalloc/malloc.cc:35:10: fatal error: > > 'override/jemalloc_compat.cc' file not found > > #include "override/jemalloc_compat.cc" > > ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > 1 error generated. > > --- malloc.o --- > > *** [malloc.o] Error code 1 > > > > make[4]: stopped in /usr/src/lib/libc > > /usr/src/lib/libc/stdlib/snmalloc/memcpy.cc:25:10: fatal error: > > 'global/memcpy.h' file not found > > #include > > ^~~~~~~~~~~~~~~~~ > > 1 error generated. > > --- memcpy.o --- > > *** [memcpy.o] Error code 1 > > This looks as if you haven=E2=80=99t got the submodule? Is there anythin= g in > contrib/snmalloc? > And that is why we can't just start using submodules. They are not automatically used. People have to do different things that need to be publicized and well documented. And there are about a half a dozen decisions that need to be made about the details of their use, many of which have no clear obvious choice for the project. Without a good plan, clear comms and good docs it will be a support nightmare. Now please stop making these passive agressive comments about submodules. All they do is demotivate me to work on the plans to adopt them. There are a lot of details, many of which need to be play tested, before we can even get a plan to adopt. The snarky comments are why I quit working on things a year ago... they don't move the ball forward and just piss people off... Warner > this is a fresh world, top of snmalloc2 branch: > > commit a5c83c69817d03943b8be982dd815c7e263d1a83 > > Author: David Chisnall > > Date: Fri Jan 21 15:13:09 2022 +0000 > > > > Initial commit of snmalloc2 in libc. > > > > anyway, I wanted to say I find the memcpy thing incredibly suspicious. > > I found one article in > > > https://github.com/microsoft/snmalloc/blob/main/docs/security/GuardedMemc= py.md > > which benches it and that made it even more suspicious. How did the > > benched memcpy look like inside? > > Perhaps you could share what you are suspicious about? I don=E2=80=99t r= eally > know how to respond to something so vague. The document you linked to ha= s > the benchmark that we used (though the graphs in it appear to be based on > an older version of the memcpy). The PR that added PowerPC tuning has so= me > additional graphs of measurements. > > If you compile the memcpy file, you can see the assembly. The C++ > provides a set of building blocks for producing efficient memcpy > implementations. The fastest on x86 is roughly: > > - A jump table of power for small sizes that do power-of-two-sized small > copies (double-word, word, half-word, and byte) to perform the copy. > - A vectorised copy for medium-sized copies using a loop of SSE copies. > - rep movsb for large copies. > > The compiler does some quite complex layout for the jump table. > > David > > > --0000000000000fdb0105f449a32b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Feb 9, 2023, 12:37 PM David Chisnall <theraven@freebsd.org> wrote:
<= /div>
On 9 Feb 2023, at 19:15, Mateusz Guzik = <mjguzik@gmail.com> wrote:
>
> it fails to build for me:
>
> /usr/src/lib/libc/stdlib/snmalloc/malloc.cc:35:10: fatal error:
> 'override/jemalloc_compat.cc' file not found
> #include "override/jemalloc_compat.cc"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 1 error generated.
> --- malloc.o ---
> *** [malloc.o] Error code 1
>
> make[4]: stopped in /usr/src/lib/libc
> /usr/src/lib/libc/stdlib/snmalloc/memcpy.cc:25:10: fatal error:
> 'global/memcpy.h' file not found
> #include <global/memcpy.h>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0^~~~~~~~~~~~~~~~~
> 1 error generated.
> --- memcpy.o ---
> *** [memcpy.o] Error code 1

This looks as if you haven=E2=80=99t got the submodule?=C2=A0 Is there anyt= hing in contrib/snmalloc?
And that is why we can't just start using subm= odules. They are not automatically used. People have to do different=C2=A0t= hings that need to be publicized and well documented. And there are about a= half a dozen decisions that need to be made about the details of their use= , many of which have no clear obvious choice for the project. Without a goo= d plan, clear comms and good docs it will be a support nightmare.=C2=A0

Now please stop making thes= e passive agressive comments about submodules. All they do is demotivate me= to work on the plans to adopt them. There are a lot of details, many of wh= ich need to be play tested, before we can even get a plan to adopt. The sna= rky comments are why I quit working on things a year ago... they don't = move the ball forward and just piss people off...
Warner=C2=A0

> this is a fresh world, top of snmalloc2 branch:
> commit a5c83c69817d03943b8be982dd815c7e263d1a83
> Author: David Chisnall <theraven@FreeBSD.org>
> Date:=C2=A0 =C2=A0Fri Jan 21 15:13:09 2022 +0000
>
>=C2=A0 =C2=A0 Initial commit of snmalloc2 in libc.
>
> anyway, I wanted to say I find the memcpy thing incredibly suspicious.=
> I found one article in
> https:= //github.com/microsoft/snmalloc/blob/main/docs/security/GuardedMemcpy.md
> which benches it and that made it even more suspicious. How did the > benched memcpy look like inside?

Perhaps you could share what you are suspicious about?=C2=A0 I don=E2=80=99= t really know how to respond to something so vague.=C2=A0 The document you = linked to has the benchmark that we used (though the graphs in it appear to= be based on an older version of the memcpy).=C2=A0 The PR that added Power= PC tuning has some additional graphs of measurements.

If you compile the memcpy file, you can see the assembly.=C2=A0 The C++ pro= vides a set of building blocks for producing efficient memcpy implementatio= ns.=C2=A0 The fastest on x86 is roughly:

=C2=A0- A jump table of power for small sizes that do power-of-two-sized sm= all copies (double-word, word, half-word, and byte) to perform the copy. =C2=A0- A vectorised copy for medium-sized copies using a loop of SSE copie= s.
=C2=A0- rep movsb for large copies.

The compiler does some quite complex layout for the jump table.

David


--0000000000000fdb0105f449a32b-- From nobody Thu Feb 9 20:14:07 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCSkT5MbJz3nZsm for ; Thu, 9 Feb 2023 20:14:09 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCSkT4rnVz3skq; Thu, 9 Feb 2023 20:14:09 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675973649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YGkETsO/J4J6CwxwAZqX4psHr0rK4whm/B0g+B7uNHo=; b=f09maRmPTMSEGYWBHNestZSzd1omaBmkNCmMf03ovAMpYmoQUj+SRnHtr9t4LHz3bsUdoo JjsL7px4gBvMj7sbd+Mcju9SD259RiYJmWCmaHg+LiQWWaxDZIb/dW6f70dkfyZGEgoLHj 0R6LA0Ae8MgA/Emnf1vrOQsIht6dTxaKbjfGBbBvn923K+rXgcxUhDKTG79EQQEOSjkUIK cuBKtaNkns38DXNKjgmuwYOMgygrAx7TyVMd+NGlFbXeiP7vJSv8cTO1gOyoV1nwDbsB3i WhnHo/wNP90s1dBmu7MzYMxnszO2+CRNELFD3WuQApe4Hfodz4CZQlTbQIhaFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675973649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YGkETsO/J4J6CwxwAZqX4psHr0rK4whm/B0g+B7uNHo=; b=mBfC6Eq9niqsdJbgorrTmKrz/RczqXTAzyttPHBgn3ZEfMoCYUG4sK8fLUViT/2P5Pp8nU r9jhm/8jNaVBRrDqo3MPSMciMTj3ZwlSnFfVxDjmPYD9rKCw4DL9c/ncCNP1Fq+gnCrpaY g02jn3I5MPY5CynxeQHS4ucsXlAa3lIf2EHhUSG68dbzUiQa50h8cSRYuYuoej4vy9xMpA k06KDvKlhu/7BUION3ymvb1EHdO0HFrTsWDNfd5tubOm/dCRFJZoGsKjzRMpbipYA0C2re if6xM2qZaY0lNVJLo+k/F7ZOz2YfsUw8ebo5PGUypTvhr+C8bqcTj8RZ+p0dkg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675973649; a=rsa-sha256; cv=none; b=hMApcyHKVnO8ylAJHs8c3nLWa+Xoc9qcEL4kPe0qLg+yWknb5+mI5DatxRH9M2kxoXNrTh eGANFdUTakrZu5bO9Zhn0NSma+nPqbsYyJU6UIPukMecEMQGHKLnh0LHdAHOEadKaSRCr+ cTY6qonX9BBGDzSA9TkZwg6kkGtHAuZ0ZzotCES/bueSkBrqOvLwbbFSJ9rm1X7IhF8Kei TVeFHMVhfN339Cb5AQKjyKCvc2+sELBdxtHfS7VzrQ/FdDo+D12wxjOOC3MAzRP2EwaPrf RbF4CB+bO8ovBet6W5M4+d5kKlPDdcnKF5BRWlhEuqI9b6SzRg6CvMk2PTjbIQ== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PCSkT3n30zWPZ; Thu, 9 Feb 2023 20:14:09 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host81-158-36-31.range81-158.btcentralplus.com [81.158.36.31]) by smtp.theravensnest.org (Postfix) with ESMTPSA id CAA161058B; Thu, 9 Feb 2023 20:14:08 +0000 (GMT) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: CFT: snmalloc as libc malloc From: David Chisnall In-Reply-To: Date: Thu, 9 Feb 2023 20:14:07 +0000 Cc: Mateusz Guzik , freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> To: Warner Losh X-Mailer: Apple Mail (2.3654.120.0.1.13) X-ThisMailContainsUnwantedMimeParts: N On 9 Feb 2023, at 19:46, Warner Losh wrote: >=20 > And that is why we can't just start using submodules. They are not = automatically used. They are not used automatically, but if we were using them then we would = have infrastructure in the build system for ensuring that they exist and = checking them out if they are not. =20 I have written some (trivial) scripts to do this in a dev container = recently for another project, the developer experience there is go to = github, hit ., hit =E2=80=98open in code space=E2=80=99, start writing = code in the browser. =20 I didn=E2=80=99t do something like that here because I didn't want this = tree to contain things that I definitely couldn=E2=80=99t upstream and = you=E2=80=99ve made it quite clear that upstreaming will require me to = do something different. > People have to do different things that need to be publicized and well = documented. As with anything involving revision control. > And there are about a half a dozen decisions that need to be made = about the details of their use, many of which have no clear obvious = choice for the project. Without a good plan, clear comms and good docs = it will be a support nightmare.=20 Something I would be happy to work on, but the message that I = consistently get is =E2=80=98we won=E2=80=99t use submodules, we are not = open to a plan to use submodules=E2=80=99. > Now please stop making these passive agressive comments about = submodules. All they do is demotivate me to work on the plans to adopt = them. There are a lot of details, many of which need to be play tested, = before we can even get a plan to adopt. The snarky comments are why I = quit working on things a year ago... they don't move the ball forward = and just piss people off... I will very happily stop making comments about submodules if there is = either: - A working group that I can participate in to propose a plan for using = them. Or even a willingness, if I write a plan for using submodules, = for it to be discussed and not rejected out of hand. - An alternative proposal that doesn=E2=80=99t have the downsides that = we currently have (for example, forcing everyone to duplicate the = history of all every LLVM version that FreeBSD ships in a git clone, = difficulty of CI testing contrib components in isolation, complex steps = to import a new version from upstream, and so on), or worse down-sides = than submodules (e.g. depending on additional tooling that doesn=E2=80=99t= integrate with other tools, impossibility of using the tooling on some = platforms, massive clone sizes, and so on) I am frustrated by the fact that the project has real problems that can = be solved by submodules, does not want to use them, and does not want to = solve them in another way either. I don=E2=80=99t particularly like = submodules, I just like the problems that are caused by not using them = even less. David From nobody Thu Feb 9 20:53:34 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCTc01SYFz3ngNH for ; Thu, 9 Feb 2023 20:53:36 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x34.google.com (mail-oa1-x34.google.com [IPv6:2001:4860:4864:20::34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCTbz6jNqz42Cj; Thu, 9 Feb 2023 20:53:35 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x34.google.com with SMTP id 586e51a60fabf-15ff0a1f735so4220257fac.5; Thu, 09 Feb 2023 12:53:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=w9IqZLORatmwNq5xtYIAdz/iwBQguJphxQ1xCd3ujfI=; b=LB3KGCYE+L/s8XddkSQ0lAAXRGpOtbrGm2hskFTcu0MbN9FsX1JAl64NUO9YUbEWyG 3vQdkRS9Nzxz+xsfNoXm9a4bnAV1Dpwwl2dm2uYrkQOrGJ3dfSIePRejbjeAYhONVKWO bR2VnSg1xbNP3kFEalN2ImUuNSRj/cKP99NqAV5gY0N9KKr5o1A39gP8iHbed0d0W3E3 hV0093eW9FsquXYfflPeM8fXaMQXyVZwRJTFe4tDZ2i533Cnj6H5t6Hlb5TLZQwcEFOw cLa4ZWjrwcTETGrtJCPpYmUVgfh8gJMfGkETK5b1WjTB1X4nmd9ZhUmQY6glXh07CeeM ebeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w9IqZLORatmwNq5xtYIAdz/iwBQguJphxQ1xCd3ujfI=; b=Kfv7QxRiUTWq2kZPgHUaWDwezQRXeqAwaUOXxVKK2WgwzkZe8x3g6ssnfhVD079PxQ VASzeyIb+ceRdwSFP3zMlNBhS731jOY+eILl1lDAqP7HpEQB6DYpgZJpILR7c/2fqmLb pufAa2ylS8msuqiw1L2KH9SXg3pZ3rjmoMEAugQNM5CG47sTSweESlBn0xeiNuKMUAFq TlDOcJErhjMZMg6fKzYea/e3wX8uzK3Zw5DkypUDE2m2pNxXNCFPAuFLdtlPw4YWfTcF BNSK0uDt6WIy18Wy2CJJMRjqFSQ/8CzoabN9kIufUtPjkSIdwp9dqSq25ugdL2Zlqfwd cnmA== X-Gm-Message-State: AO0yUKXC64WqcULd816LwgkWuChIndelvUVc54F2N6pkO+ZOnYSIABWt o7SSRb37SijPIwp7lv7HA6k6FZHqonLJ9xPwam9WoMkd X-Google-Smtp-Source: AK7set9u3oE24Dzb10sJ7sEGwRszv/ivCDlEcTeAxuBcTjB6x193AruTFVrE9C7yB2FMbmR82hCgDtvNH1UcqN84OL8= X-Received: by 2002:a05:6870:1257:b0:16a:9099:3868 with SMTP id 23-20020a056870125700b0016a90993868mr784833oao.81.1675976014964; Thu, 09 Feb 2023 12:53:34 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:6c92:0:b0:4b3:d953:974c with HTTP; Thu, 9 Feb 2023 12:53:34 -0800 (PST) In-Reply-To: References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> From: Mateusz Guzik Date: Thu, 9 Feb 2023 21:53:34 +0100 Message-ID: Subject: Re: CFT: snmalloc as libc malloc To: David Chisnall Cc: freebsd-hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4PCTbz6jNqz42Cj X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 2/9/23, David Chisnall wrote: > On 9 Feb 2023, at 19:15, Mateusz Guzik wrote: >> >> it fails to build for me: >> >> /usr/src/lib/libc/stdlib/snmalloc/malloc.cc:35:10: fatal error: >> 'override/jemalloc_compat.cc' file not found >> #include "override/jemalloc_compat.cc" >> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> 1 error generated. >> --- malloc.o --- >> *** [malloc.o] Error code 1 >> >> make[4]: stopped in /usr/src/lib/libc >> /usr/src/lib/libc/stdlib/snmalloc/memcpy.cc:25:10: fatal error: >> 'global/memcpy.h' file not found >> #include >> ^~~~~~~~~~~~~~~~~ >> 1 error generated. >> --- memcpy.o --- >> *** [memcpy.o] Error code 1 > > This looks as if you haven=E2=80=99t got the submodule? Is there anythin= g in > contrib/snmalloc? > indeed, a pilot error >> anyway, I wanted to say I find the memcpy thing incredibly suspicious. >> I found one article in >> https://github.com/microsoft/snmalloc/blob/main/docs/security/GuardedMem= cpy.md >> which benches it and that made it even more suspicious. How did the >> benched memcpy look like inside? > > Perhaps you could share what you are suspicious about? I don=E2=80=99t r= eally know > how to respond to something so vague. The document you linked to has the > benchmark that we used (though the graphs in it appear to be based on an > older version of the memcpy). The PR that added PowerPC tuning has some > additional graphs of measurements. > > If you compile the memcpy file, you can see the assembly. The C++ provid= es > a set of building blocks for producing efficient memcpy implementations. First and foremost, perhaps I should clear up that I have no opinion on snmalloc or it replacing jemalloc. I'm here only about the memcpy thing. Inspecting assembly is what I intended to do, but got the compilation probl= em. So, as someone who worked on memcpy previously, I note the variant currently implemented in libc is pessimal for sizes > 16 bytes because it does not use SIMD. I do have plans to rectify that, but ENOTIME. I also note CPUs are incredibly picky when it comes to starting point of the routine. The officialy recommended alignment of 16 bytes is basically a tradeoff between total binary size and performance. Should you move the routine at different 16 bytes intervals you will easily find big variation in performance, depending on how big of an alignment you ended up with. I tried to compile the benchmark but got bested by c++. I don't know the lang and I don't want to fight it. $ pwd /usr/src/contrib/snmalloc/src $ c++ -I. test/perf/memcpy/memcpy.cc [snip] ./snmalloc/global/../backend/../backend_helpers/../mem/../ds_core/bits.h:57= :26: error: no template named 'is_integral_v' in namespace 'std'; did you mean 'is_integral'? static_assert(std::is_integral_v, "Type must be integral"); ~~~~~^~~~~~~~~~~~~ is_integral and tons of other errors. I did buildworld + installworld. I'm trying to say that if you end up with different funcs depending on bounds checking and you only align them to 16 bytes, the benchmark is most likely inaccurate if only for this reason. > The fastest on x86 is roughly: > > - A jump table of power for small sizes that do power-of-two-sized small > copies (double-word, word, half-word, and byte) to perform the copy. Last I checked this was not true. The last slow thing to do was to branch on few sizes and handle them with overlapping stores. Roughly what memcpy in libc is doing now. Jump table aside, you got all sizes "spelled out", that is just avoidable impact on icache. While overlapping stores do come with some penalty, it is cheaper than the above combined. I don't have numbers nor bench code handy, but if you insist on contesting the above I'll be glad to provide them. > - A vectorised copy for medium-sized copies using a loop of SSE copies. Depends on what you mean by medium and which simd instructions, but yes, they should be used here. > - rep movsb for large copies. There are still old cpus here and there which don't support ERMS. They are very negatively impacted the above and should roll with rep stosq instead. I see the code takes some care to align the target buffer. That's good, but not necessary on CPUs with FSRM. All that said, I have hard time believing the impact of bounds checking on short copies is around 20% or so. The code to do it looks super slow (not that I know to do better). People like to claim short sizes are inlined by the compiler, but that's only true if the size is known at compilation time, which it often is not. Then they claim they are rare. To illustrate why that's bogus, here is clang 15 compiling vfs_cache.c: value ------------- Distribution ------------- count -1 | 0 0 |@ 19758 1 |@@@@@@@@ 218420 2 |@@ 67670 4 |@@@@ 103914 8 |@@@@@@@@@@@ 301157 16 |@@@@@@@@@@ 293812 32 |@@ 57954 64 |@ 23818 128 | 13344 256 |@ 18507 512 | 6342 1024 | 1710 2048 | 627 4096 | 398 8192 | 34 16384 | 10 32768 | 6 65536 | 7 131072 | 4 262144 | 1 524288 | 1 1048576 | 0 obtained with this bad boy: dtrace -n 'pid$target::memcpy:entry { @ =3D quantize(arg2); }' -c "cc -target x86_64-unknown-freebsd14.0 --sysroot=3D/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe -fno-strict-aliasing -g -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD -MF.depend.vfs_cache.o -MTvfs_cache.o -fdebug-prefix-map=3D./machine=3D/usr/src/sys/amd64/include -fdebug-prefix-map=3D./x86=3D/usr/src/sys/x86/include -fdebug-prefix-map=3D./i386=3D/usr/src/sys/i386/include -mcmodel=3Dkernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=3D__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error=3Dtautological-compare -Wno-error=3Dempty-body -Wno-error=3Dparentheses-equality -Wno-error=3Dunused-function -Wno-error=3Dpointer-sign -Wno-error=3Dshift-negative-value -Wno-address-of-packed-member -Wno-error=3Darray-parameter -Wno-error=3Ddeprecated-non-prototype -Wno-error=3Dstrict-prototypes -Wno-error=3Dunused-but-set-variable -Wno-format-zero-length -mno-aes -mno-avx -std=3Diso9899:1999 -Werror /usr/src/sys/kern/vfs_cache.c" tl;dr if this goes in, the fate of memcpy thing will need to be handled separtely --=20 Mateusz Guzik From nobody Thu Feb 9 21:34:38 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCVWQ1TpWz3nmGB for ; Thu, 9 Feb 2023 21:34:42 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x22f.google.com (mail-oi1-x22f.google.com [IPv6:2607:f8b0:4864:20::22f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCVWP1kZwz4DRf; Thu, 9 Feb 2023 21:34:41 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="edFBGQZ/"; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::22f as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-x22f.google.com with SMTP id bx13so2817359oib.13; Thu, 09 Feb 2023 13:34:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=CSekNfIR/yaonlKxBB9st1/jgtV/qVcymxkTpP8fPNI=; b=edFBGQZ/0Dv1CoZlhs6K7fnfsDnXGrYleRfF+ANmbTVG0e9kpN9UqcKn2LftIy6uql /wOuwmCGTz0SPpWtb5lWou1WoUq/OeemnMDVB0wdPy8bTp2uEq571lIQqiAMm6XavuYa 4xM/r3mIyLkZb388VWI0piWZ5YOks+KGPfV1/p8GGXUxbiKy2hmlG8en5NXBkY42vQep 23m6cih5cBPi18y2A8NOQdTAoPkOX3/TirW/cT/Ltll9Rmn1/Mec3UfWfL7c6iBKi5yD FL+RNixFdztrcVAXBKLv4jiaWFhHk3w3w0VrSLb+FnShdGewpunjUTnYTFiHGPdtR2b5 lqFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CSekNfIR/yaonlKxBB9st1/jgtV/qVcymxkTpP8fPNI=; b=N3c1+nsQPcTewMW7Lstik8yGRT4oS0nYbgf5T2Nb0kqTg+y4DaKgvUb6pqVA7iZWG3 ueqToPSk62nZh6v7KiYAnS1ZnM6xS16C6PsO2iwLFrVtWxWI+8zYldvEnfPM8jfOSAZ+ YESnuVd4oy+Bd3ntwe1c1I6U2jhxRxcJR9xSgbTqyYTleTNqf06y9y+w8OxzXzFk1A4W mE5AMQHBlmEW2xJcOHXsOygQBFCU5SO9SesxfZtRSjapWtV1DTwVjdZbo/hj6xp3l4vb trjFz1Xla9U0Eo0gO/X1YH27EbEa783Wey/YRM62oJdsVrNSU47HIKx1KTrb8LqcN9gT fBsg== X-Gm-Message-State: AO0yUKWZRS0/pT9ICXbVU0Prw3o+O7aoR2ZpocfHOlM1mBib3ZgI7+dR sx0klGm1GmhnAcXbJSzD+n1boyE9qjUHN8+KtXbvs1M8 X-Google-Smtp-Source: AK7set8YI8dIgu1ICNdroyMgjoiMCqwUDzYMzbR1Jpk5OKqc6fJF9ZlYqvGnG73H84TCMppHIl64cEq3N8KoE9B9ql0= X-Received: by 2002:aca:5905:0:b0:37a:ca27:ae3d with SMTP id n5-20020aca5905000000b0037aca27ae3dmr650254oib.159.1675978479457; Thu, 09 Feb 2023 13:34:39 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:6c92:0:b0:4b3:d953:974c with HTTP; Thu, 9 Feb 2023 13:34:38 -0800 (PST) In-Reply-To: References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> From: Mateusz Guzik Date: Thu, 9 Feb 2023 22:34:38 +0100 Message-ID: Subject: Re: CFT: snmalloc as libc malloc To: David Chisnall Cc: freebsd-hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-2.90 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.96)[-0.957]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.06)[0.058]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::22f:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4PCVWP1kZwz4DRf X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N The memcpy debacle aside, I can confirm that single-threaded the new malloc does appear faster in naive tests from will-it-scale: $ cpuset -l 10,80,82 -- ./malloc2_threads -n -t 2 testcase:malloc/free of 1kB before: min:97812514 max:97849385 total:195661899 min:97819901 max:97857131 total:195677032 min:97789741 max:97833562 total:195623303 after: min:115613762 max:124855002 total:240468764 min:115636562 max:124807148 total:240443710 min:115778776 max:124784220 total:240562996 that said, if anyone is to performed a serious test, the stock memcpy needs to be used to rule it out as a factor. The one shipped with snmalloc will happen to be faster for certain sizes and that may skew whatever evaluation -- that speed increase (and in fact higher) is achievable without snmalloc. On 2/9/23, Mateusz Guzik wrote: > On 2/9/23, David Chisnall wrote: >> On 9 Feb 2023, at 19:15, Mateusz Guzik wrote: >>> >>> it fails to build for me: >>> >>> /usr/src/lib/libc/stdlib/snmalloc/malloc.cc:35:10: fatal error: >>> 'override/jemalloc_compat.cc' file not found >>> #include "override/jemalloc_compat.cc" >>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> 1 error generated. >>> --- malloc.o --- >>> *** [malloc.o] Error code 1 >>> >>> make[4]: stopped in /usr/src/lib/libc >>> /usr/src/lib/libc/stdlib/snmalloc/memcpy.cc:25:10: fatal error: >>> 'global/memcpy.h' file not found >>> #include >>> ^~~~~~~~~~~~~~~~~ >>> 1 error generated. >>> --- memcpy.o --- >>> *** [memcpy.o] Error code 1 >> >> This looks as if you haven=E2=80=99t got the submodule? Is there anythi= ng in >> contrib/snmalloc? >> > > indeed, a pilot error > >>> anyway, I wanted to say I find the memcpy thing incredibly suspicious. >>> I found one article in >>> https://github.com/microsoft/snmalloc/blob/main/docs/security/GuardedMe= mcpy.md >>> which benches it and that made it even more suspicious. How did the >>> benched memcpy look like inside? >> >> Perhaps you could share what you are suspicious about? I don=E2=80=99t = really >> know >> how to respond to something so vague. The document you linked to has th= e >> benchmark that we used (though the graphs in it appear to be based on an >> older version of the memcpy). The PR that added PowerPC tuning has some >> additional graphs of measurements. >> >> If you compile the memcpy file, you can see the assembly. The C++ >> provides >> a set of building blocks for producing efficient memcpy implementations. > > First and foremost, perhaps I should clear up that I have no opinion > on snmalloc or it replacing jemalloc. I'm here only about the memcpy > thing. > > Inspecting assembly is what I intended to do, but got the compilation > problem. > > So, as someone who worked on memcpy previously, I note the variant > currently implemented in libc is pessimal for sizes > 16 bytes because > it does not use SIMD. I do have plans to rectify that, but ENOTIME. > > I also note CPUs are incredibly picky when it comes to starting point > of the routine. The officialy recommended alignment of 16 bytes is > basically a tradeoff between total binary size and performance. Should > you move the routine at different 16 bytes intervals you will easily > find big variation in performance, depending on how big of an > alignment you ended up with. > > I tried to compile the benchmark but got bested by c++. I don't know > the lang and I don't want to fight it. > > $ pwd > /usr/src/contrib/snmalloc/src > $ c++ -I. test/perf/memcpy/memcpy.cc > [snip] > ./snmalloc/global/../backend/../backend_helpers/../mem/../ds_core/bits.h:= 57:26: > error: no template named 'is_integral_v' in namespace 'std'; did you > mean 'is_integral'? > static_assert(std::is_integral_v, "Type must be integral"); > ~~~~~^~~~~~~~~~~~~ > is_integral > > and tons of other errors. I did buildworld + installworld. > > I'm trying to say that if you end up with different funcs depending on > bounds checking and you only align them to 16 bytes, the benchmark is > most likely inaccurate if only for this reason. > >> The fastest on x86 is roughly: >> >> - A jump table of power for small sizes that do power-of-two-sized smal= l >> copies (double-word, word, half-word, and byte) to perform the copy. > > Last I checked this was not true. The last slow thing to do was to > branch on few sizes and handle them with overlapping stores. Roughly > what memcpy in libc is doing now. > > Jump table aside, you got all sizes "spelled out", that is just > avoidable impact on icache. While overlapping stores do come with some > penalty, it is cheaper than the above combined. > > I don't have numbers nor bench code handy, but if you insist on > contesting the above I'll be glad to provide them. > >> - A vectorised copy for medium-sized copies using a loop of SSE copies. > > Depends on what you mean by medium and which simd instructions, but > yes, they should be used here. > >> - rep movsb for large copies. > > There are still old cpus here and there which don't support ERMS. They > are very negatively impacted the above and should roll with rep stosq > instead. > > I see the code takes some care to align the target buffer. That's > good, but not necessary on CPUs with FSRM. > > All that said, I have hard time believing the impact of bounds > checking on short copies is around 20% or so. The code to do it looks > super slow (not that I know to do better). > > People like to claim short sizes are inlined by the compiler, but > that's only true if the size is known at compilation time, which it > often is not. Then they claim they are rare. > > To illustrate why that's bogus, here is clang 15 compiling vfs_cache.c: > value ------------- Distribution ------------- count > -1 | 0 > 0 |@ 19758 > 1 |@@@@@@@@ 218420 > 2 |@@ 67670 > 4 |@@@@ 103914 > 8 |@@@@@@@@@@@ 301157 > 16 |@@@@@@@@@@ 293812 > 32 |@@ 57954 > 64 |@ 23818 > 128 | 13344 > 256 |@ 18507 > 512 | 6342 > 1024 | 1710 > 2048 | 627 > 4096 | 398 > 8192 | 34 > 16384 | 10 > 32768 | 6 > 65536 | 7 > 131072 | 4 > 262144 | 1 > 524288 | 1 > 1048576 | 0 > > obtained with this bad boy: > dtrace -n 'pid$target::memcpy:entry { @ =3D quantize(arg2); }' -c "cc > -target x86_64-unknown-freebsd14.0 > --sysroot=3D/usr/obj/usr/src/amd64.amd64/tmp > -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe > -fno-strict-aliasing -g -nostdinc -I. -I/usr/src/sys > -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt > -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h > -fno-common -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer > -MD -MF.depend.vfs_cache.o -MTvfs_cache.o > -fdebug-prefix-map=3D./machine=3D/usr/src/sys/amd64/include > -fdebug-prefix-map=3D./x86=3D/usr/src/sys/x86/include > -fdebug-prefix-map=3D./i386=3D/usr/src/sys/i386/include -mcmodel=3Dkernel > -mno-red-zone -mno-mmx -mno-sse -msoft-float > -fno-asynchronous-unwind-tables -ffreestanding -fwrapv > -fstack-protector -Wall -Wnested-externs -Wstrict-prototypes > -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef > -Wno-pointer-sign -D__printf__=3D__freebsd_kprintf__ > -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas > -Wno-error=3Dtautological-compare -Wno-error=3Dempty-body > -Wno-error=3Dparentheses-equality -Wno-error=3Dunused-function > -Wno-error=3Dpointer-sign -Wno-error=3Dshift-negative-value > -Wno-address-of-packed-member -Wno-error=3Darray-parameter > -Wno-error=3Ddeprecated-non-prototype -Wno-error=3Dstrict-prototypes > -Wno-error=3Dunused-but-set-variable -Wno-format-zero-length -mno-aes > -mno-avx -std=3Diso9899:1999 -Werror /usr/src/sys/kern/vfs_cache.c" > > tl;dr if this goes in, the fate of memcpy thing will need to be > handled separtely > -- > Mateusz Guzik > --=20 Mateusz Guzik From nobody Thu Feb 9 21:38:40 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCVc25q7Fz3nmdn for ; Thu, 9 Feb 2023 21:38:42 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCVc25Kzlz4GLL; Thu, 9 Feb 2023 21:38:42 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675978722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SP/1WlxvQylAiIe3tHKGFQyP6mRWp2V4gr/XfNuvV5o=; b=TBef7PTOSzrKghx+ffG/lpRZJHe83OBsLsEe6BYOLsdmfx0Nq4nSPQX2Eu1zj0HCmN3UIu bCNdrRg9lfs5hKwqo8qAc8K9OkWmpV3g0kLThsPKY2+B6jW9kIylzKFDaTCpRvqnJ/VKT3 TRaSwQtSZrnglLw6GKlEUBzxZbCLJ/bMytkKaP7KT5fewX5dYSFC57c8WKQwhP/DX+9pty SDOHHI6jKsqOPECqkU0AdTnewJ3zxunHzc1hDqff2Qo5w04BCSr+jDBiN6RcPPgCdcltCh X/6k3ccbpU4fqXm7f7nWJQxOqYssj6HLsRPSUcXRyJheEEwY1l+h3x+J/wyRog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1675978722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SP/1WlxvQylAiIe3tHKGFQyP6mRWp2V4gr/XfNuvV5o=; b=QVNZ/SyIkPMxofUvd7lBwomf7h/iq7L2MEj8J9zMYbPjh1WsuySXGE6Ac/AEFf1bdHJ9GQ QzoHpAXdn0T8LLdySnlpUuQuH9q6f2abiGGn1T1Z+EgVbeIllfscuJ2Q0KrqnxIT0ydOx0 Y1SYSTHHdnJloi7pOm+cWq3xBzak2JDa4K7GFvPnY2gY5VJSOyI2TVTYMeJhlSvmQGQzNM jkU+7uIVw5eTA8MIkqi/nk8VApYyCyrqRnsapHpwZ3CDVbWwUhtcw3bvrjzjIXAULhwYNr uQO7tP5wRDgJ07ZLYQyFh5/n5/Eei9C2mUj8sX2bnd62d8oS6+ExvMx4aM1YcQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1675978722; a=rsa-sha256; cv=none; b=Tpzzine3z9UlKKjDBgdfXoBgq+MGhYwGTB1EeNWOYOn27UCrcwZ+hJG7+LhyIg0oVopmKG 9zkuVwPJ+sj453GPpAxGUPqyXoOvhDXbXhh0x1v6yREzGxh8+l/qfpOm9WZjYOnfhRc2aZ yMd6HYwpAMxXKQDHKcCVbXgLEFEBsiP4aOBBQP8DDto01oUkIH0bm6vQwwwl4R/KGj07UD gOXhY+hpj2Rkzcv9/GLHXpk00Qvbz5gysyQ33Y24e5tqZai+dzeKKmpRLkjqw55uHey49x yZcuvlCqTxa8i/bfpIqVVPEzUHXSyzZFbJQgpUiPoISYSwhVqTFgk9UIDReEyA== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PCVc24GYmzXQW; Thu, 9 Feb 2023 21:38:42 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host81-158-36-31.range81-158.btcentralplus.com [81.158.36.31]) by smtp.theravensnest.org (Postfix) with ESMTPSA id B15E91058C; Thu, 9 Feb 2023 21:38:41 +0000 (GMT) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: CFT: snmalloc as libc malloc From: David Chisnall In-Reply-To: Date: Thu, 9 Feb 2023 21:38:40 +0000 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: <2836EF0C-8B4B-441C-927B-3796457A3865@FreeBSD.org> References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> To: Mateusz Guzik X-Mailer: Apple Mail (2.3654.120.0.1.13) X-ThisMailContainsUnwantedMimeParts: N On 9 Feb 2023, at 20:53, Mateusz Guzik wrote: >=20 > First and foremost, perhaps I should clear up that I have no opinion > on snmalloc or it replacing jemalloc. I'm here only about the memcpy > thing. >=20 > Inspecting assembly is what I intended to do, but got the compilation = problem. If you just want to look at the memcpy, you might do better using the = upstream (CMake) build system, which builds a shim library that you can = disassemble and look at. > So, as someone who worked on memcpy previously, I note the variant > currently implemented in libc is pessimal for sizes > 16 bytes because > it does not use SIMD. I do have plans to rectify that, but ENOTIME. That=E2=80=99s one of the benefits of the C++ implementation. We were = able to try a few dozen variations in a morning. Writing a single one = of those in assembly would take a similar amount of time. We were inspired by the automemcpy paper, which demonstrated that you = can write memcpy implementations tuned to specific workloads in = higher-level languages and get better performance. > I also note CPUs are incredibly picky when it comes to starting point > of the routine. The officialy recommended alignment of 16 bytes is > basically a tradeoff between total binary size and performance. Should > you move the routine at different 16 bytes intervals you will easily > find big variation in performance, depending on how big of an > alignment you ended up with. Yes, that=E2=80=99s what our test does. It does a large number of = different copies with different sizes and alignments. =20 > I tried to compile the benchmark but got bested by c++. I don't know > the lang and I don't want to fight it. >=20 > $ pwd > /usr/src/contrib/snmalloc/src > $ c++ -I. test/perf/memcpy/memcpy.cc > [snip] > = ./snmalloc/global/../backend/../backend_helpers/../mem/../ds_core/bits.h:5= 7:26: > error: no template named 'is_integral_v' in namespace 'std'; did you > mean 'is_integral'? > static_assert(std::is_integral_v, "Type must be integral"); > ~~~~~^~~~~~~~~~~~~ > is_integral I think the only thing missing is -std=3Dc++17. > I'm trying to say that if you end up with different funcs depending on > bounds checking and you only align them to 16 bytes, the benchmark is > most likely inaccurate if only for this reason. I=E2=80=99m not sure I follow this, sorry. >> The fastest on x86 is roughly: >>=20 >> - A jump table of power for small sizes that do power-of-two-sized = small >> copies (double-word, word, half-word, and byte) to perform the copy. >=20 > Last I checked this was not true. The last slow thing to do was to > branch on few sizes and handle them with overlapping stores. Roughly > what memcpy in libc is doing now. >=20 > Jump table aside, you got all sizes "spelled out", that is just > avoidable impact on icache. While overlapping stores do come with some > penalty, it is cheaper than the above combined. They=E2=80=99re not all spelled out, because a lot of them are subsets = of the others. The compiler does a *very* good job of optimising this. = The generated assembly was a lot more complex than anything I could = write by hand. > I don't have numbers nor bench code handy, but if you insist on > contesting the above I'll be glad to provide them. >=20 >> - A vectorised copy for medium-sized copies using a loop of SSE = copies. >=20 > Depends on what you mean by medium and which simd instructions, but > yes, they should be used here. This could probably do with more tuning, but all of this is configurable = with a couple of template parameters. If it=E2=80=99s useful to have = more variants for different microarchitectures then it=E2=80=99s a = trivial tweak to generate them. >> - rep movsb for large copies. >=20 > There are still old cpus here and there which don't support ERMS. They > are very negatively impacted the above and should roll with rep stosq > instead. We tested on some microarchitectures without FRMS but didn=E2=80=99t = compare with rep stosq as an alternative. The rep movsb is inline = assembly = (https://github.com/microsoft/snmalloc/blob/4370a23f3e5f1d1d06967b1e0d6162= bf1ee81196/src/snmalloc/global/memcpy.h#L351) so it would be quite easy = to compare. Very happy to take patches that make things faster! PowerPC doesn=E2=80=99t have an equivalent of rep movsb, so the PowerPC = version doesn=E2=80=99t have an equivalent codepath. Each tuned version is a dozen lines of code (plus a lot of comments to = explain *why* it does things a particular way), so it=E2=80=99s very = easy to add different versions with different tuning. =20 > I see the code takes some care to align the target buffer. That's > good, but not necessary on CPUs with FSRM. It doesn=E2=80=99t hurt though, on the sizes where it=E2=80=99s actually = beneficial to use the rep sequence the cost of alignment is negligible. > All that said, I have hard time believing the impact of bounds > checking on short copies is around 20% or so. The code to do it looks > super slow (not that I know to do better). The last time I looked at the disassembly, the code for the bounds check = touched three cache lines and has two branches. It fits well in a = superscalar pipeline so adds a handful of cycles. This is basically in = the noise for large copies but is double-digit percentage overhead for = small ones. > People like to claim short sizes are inlined by the compiler, but > that's only true if the size is known at compilation time, which it > often is not. Then they claim they are rare. I agree. > To illustrate why that's bogus, here is clang 15 compiling = vfs_cache.c: > value ------------- Distribution ------------- count > -1 | 0 > 0 |@ 19758 > 1 |@@@@@@@@ 218420 > 2 |@@ 67670 > 4 |@@@@ 103914 > 8 |@@@@@@@@@@@ 301157 > 16 |@@@@@@@@@@ 293812 > 32 |@@ 57954 > 64 |@ 23818 > 128 | 13344 > 256 |@ 18507 > 512 | 6342 > 1024 | 1710 > 2048 | 627 > 4096 | 398 > 8192 | 34 > 16384 | 10 > 32768 | 6 > 65536 | 7 > 131072 | 4 > 262144 | 1 > 524288 | 1 > 1048576 | 0 >=20 > obtained with this bad boy: > dtrace -n 'pid$target::memcpy:entry { @ =3D quantize(arg2); }' -c "cc > -target x86_64-unknown-freebsd14.0 > --sysroot=3D/usr/obj/usr/src/amd64.amd64/tmp > -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe > -fno-strict-aliasing -g -nostdinc -I. -I/usr/src/sys > -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt > -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h > -fno-common -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer > -MD -MF.depend.vfs_cache.o -MTvfs_cache.o > -fdebug-prefix-map=3D./machine=3D/usr/src/sys/amd64/include > -fdebug-prefix-map=3D./x86=3D/usr/src/sys/x86/include > -fdebug-prefix-map=3D./i386=3D/usr/src/sys/i386/include = -mcmodel=3Dkernel > -mno-red-zone -mno-mmx -mno-sse -msoft-float > -fno-asynchronous-unwind-tables -ffreestanding -fwrapv > -fstack-protector -Wall -Wnested-externs -Wstrict-prototypes > -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef > -Wno-pointer-sign -D__printf__=3D__freebsd_kprintf__ > -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas > -Wno-error=3Dtautological-compare -Wno-error=3Dempty-body > -Wno-error=3Dparentheses-equality -Wno-error=3Dunused-function > -Wno-error=3Dpointer-sign -Wno-error=3Dshift-negative-value > -Wno-address-of-packed-member -Wno-error=3Darray-parameter > -Wno-error=3Ddeprecated-non-prototype -Wno-error=3Dstrict-prototypes > -Wno-error=3Dunused-but-set-variable -Wno-format-zero-length = -mno-aes > -mno-avx -std=3Diso9899:1999 -Werror /usr/src/sys/kern/vfs_cache.c=E2=80= =9D That=E2=80=99s a really helpful datapoint, thanks! That distribution = matches roughly what we=E2=80=99ve seen - the number of zero-byte memcpy = calls surprised me the first time I saw it, but the automemcpy paper = reported something similar. If you can also extract the alignments from these then it would be = fairly easy to modify the benchmark program to use those distributions. My original version extended the FreeBSD assembly memcpy with a call to = a function that did the bounds checks, but that had *much* worse = performance. We could insert assembly to do the bounds checks, but = that=E2=80=99s hard to update if we change the metadata layout in = malloc. We could modify the ifunc resolvers to use the current memcpy if you = don=E2=80=99t care about bounds checks. I did a tiny amount of testing = (mostly clang and lld) in this configuration and it was slower than = using the snmalloc memcpy, but I didn=E2=80=99t do enough benchmarking = to be confident of that. Hence the CFT. David From nobody Thu Feb 9 22:07:45 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCWFd60D0z3nrmP for ; Thu, 9 Feb 2023 22:07:49 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x2a.google.com (mail-oa1-x2a.google.com [IPv6:2001:4860:4864:20::2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCWFd3HdMz4LGr; Thu, 9 Feb 2023 22:07:49 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x2a.google.com with SMTP id 586e51a60fabf-16aca2628c6so2222535fac.7; Thu, 09 Feb 2023 14:07:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=HLqBNCcGg6OMsRZUjjXYbQ3cTYQGpJoZs+s2jA67Lb4=; b=E6A2Gs6s6iirgR+fEuU9FWkAI1N4oqhW6XCoaGISiGrbaJ06iSpEAnO13Qsgr0kl8w b+1nM3mfF+oMm9Crz7wd9xpwLqb6TxuhX8EZ+AeIxpjmY7rol2QOo/7hqP8gs+q2PIEa xmSqcc2xCIuGu17VgfQda274J81d07Ci8HP1h7k0Gpa7b5S+M0eykO/kJ+hxRldA0U0Z 3RrNOSbopfxb/9e4TdYuDKFpl+lPj02LHBOgizTDYcwlY9QmLmDHChxrD3VJE7TNi/bS WQ0NcmKLDPXlI58Qrh4IyXaCS5izyd1yydg9xYjsJu4NKe2RDHfKWIMadN3C81LRq5Eh K4fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HLqBNCcGg6OMsRZUjjXYbQ3cTYQGpJoZs+s2jA67Lb4=; b=v/YYZGWsGOLOB13Ggqs271jCBzRZBqsWZecRhEYwYAvWRi0ET9+Kr9IxuJ9HchkuWH v/aySeGD6hXYGIiH1X8uld2fjWXEojKe/Cm05nYN92cKEduTnGX4IyeRC2EscySgPb/G 0IY8HZmIsJJBXLr5wifEDnWdOnELlsIHr6V4fGvQBHcvdlAcNZ4FlntpiArkU2Jec536 6Vz0PPiBiWHqGZ/uWiZ/zQweDYrE3W3hvJItUCSAZ11Riv6Vgk8tpj9QNzlzKt7R7RhE eP6tERRKMUNGglq/Piep8D5XRtMf9P+OUZk78mx478DhCvjM8fnojYS2LIAUiQRi3u3c OpBw== X-Gm-Message-State: AO0yUKV15T0c1KSFEgrMDO3HTZ8YAJtD7tXarZqhR1tGyf59nUMnKvb2 WdT49JjTmiJeNoxKYMc8tHnNGJMTtr3/A9QAGExzkuTq X-Google-Smtp-Source: AK7set/ydB6Rqy5dGx+/QSDSBZkYG/buYsOmiTBEi2ULnHrYChktEpiJxLxMkva9wR0lOd2WVMWDXesh1b2qPZQ1+So= X-Received: by 2002:a05:6870:1257:b0:16a:9099:3868 with SMTP id 23-20020a056870125700b0016a90993868mr808877oao.81.1675980465905; Thu, 09 Feb 2023 14:07:45 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:6c92:0:b0:4b3:d953:974c with HTTP; Thu, 9 Feb 2023 14:07:45 -0800 (PST) In-Reply-To: <2836EF0C-8B4B-441C-927B-3796457A3865@FreeBSD.org> References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> <2836EF0C-8B4B-441C-927B-3796457A3865@FreeBSD.org> From: Mateusz Guzik Date: Thu, 9 Feb 2023 23:07:45 +0100 Message-ID: Subject: Re: CFT: snmalloc as libc malloc To: David Chisnall Cc: freebsd-hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4PCWFd3HdMz4LGr X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N I'm going to write a proper reply some time this week. For now what matters is that this CFT mixes up different things: - snmalloc - memcpy as shipped with snmalloc - bound checks for said memcpy Seeing as stock memcpy is deficient and the one provided by snmalloc is not a factor when deciding whether to switch to snmalloc, imo this should be reposted. I also note that at least on head stock jemalloc ships with debug, which again people run with by default unless they explicitly disabled it. So a better CFT would include instructions: - how to make sure jemalloc does not have debug - how to build snmalloc without affecting memcpy (in fact, it would be best if the branch for now shipped *without* custom memcpy so that one does not have to check what hapepned) frankly i don't know how to check the jemalloc thing, but I can spend some time to find out if that helps On 2/9/23, David Chisnall wrote: > On 9 Feb 2023, at 20:53, Mateusz Guzik wrote: >> >> First and foremost, perhaps I should clear up that I have no opinion >> on snmalloc or it replacing jemalloc. I'm here only about the memcpy >> thing. >> >> Inspecting assembly is what I intended to do, but got the compilation >> problem. > > If you just want to look at the memcpy, you might do better using the > upstream (CMake) build system, which builds a shim library that you can > disassemble and look at. > >> So, as someone who worked on memcpy previously, I note the variant >> currently implemented in libc is pessimal for sizes > 16 bytes because >> it does not use SIMD. I do have plans to rectify that, but ENOTIME. > > That=E2=80=99s one of the benefits of the C++ implementation. We were ab= le to try a > few dozen variations in a morning. Writing a single one of those in > assembly would take a similar amount of time. > > We were inspired by the automemcpy paper, which demonstrated that you can > write memcpy implementations tuned to specific workloads in higher-level > languages and get better performance. > >> I also note CPUs are incredibly picky when it comes to starting point >> of the routine. The officialy recommended alignment of 16 bytes is >> basically a tradeoff between total binary size and performance. Should >> you move the routine at different 16 bytes intervals you will easily >> find big variation in performance, depending on how big of an >> alignment you ended up with. > > Yes, that=E2=80=99s what our test does. It does a large number of differ= ent copies > with different sizes and alignments. > >> I tried to compile the benchmark but got bested by c++. I don't know >> the lang and I don't want to fight it. >> >> $ pwd >> /usr/src/contrib/snmalloc/src >> $ c++ -I. test/perf/memcpy/memcpy.cc >> [snip] >> ./snmalloc/global/../backend/../backend_helpers/../mem/../ds_core/bits.h= :57:26: >> error: no template named 'is_integral_v' in namespace 'std'; did you >> mean 'is_integral'? >> static_assert(std::is_integral_v, "Type must be integral"); >> ~~~~~^~~~~~~~~~~~~ >> is_integral > > I think the only thing missing is -std=3Dc++17. > >> I'm trying to say that if you end up with different funcs depending on >> bounds checking and you only align them to 16 bytes, the benchmark is >> most likely inaccurate if only for this reason. > > I=E2=80=99m not sure I follow this, sorry. > >>> The fastest on x86 is roughly: >>> >>> - A jump table of power for small sizes that do power-of-two-sized smal= l >>> copies (double-word, word, half-word, and byte) to perform the copy. >> >> Last I checked this was not true. The last slow thing to do was to >> branch on few sizes and handle them with overlapping stores. Roughly >> what memcpy in libc is doing now. >> >> Jump table aside, you got all sizes "spelled out", that is just >> avoidable impact on icache. While overlapping stores do come with some >> penalty, it is cheaper than the above combined. > > They=E2=80=99re not all spelled out, because a lot of them are subsets of= the > others. The compiler does a *very* good job of optimising this. The > generated assembly was a lot more complex than anything I could write by > hand. > >> I don't have numbers nor bench code handy, but if you insist on >> contesting the above I'll be glad to provide them. >> >>> - A vectorised copy for medium-sized copies using a loop of SSE copies. >> >> Depends on what you mean by medium and which simd instructions, but >> yes, they should be used here. > > This could probably do with more tuning, but all of this is configurable > with a couple of template parameters. If it=E2=80=99s useful to have mor= e variants > for different microarchitectures then it=E2=80=99s a trivial tweak to gen= erate > them. > >>> - rep movsb for large copies. >> >> There are still old cpus here and there which don't support ERMS. They >> are very negatively impacted the above and should roll with rep stosq >> instead. > > We tested on some microarchitectures without FRMS but didn=E2=80=99t comp= are with > rep stosq as an alternative. The rep movsb is inline assembly > (https://github.com/microsoft/snmalloc/blob/4370a23f3e5f1d1d06967b1e0d616= 2bf1ee81196/src/snmalloc/global/memcpy.h#L351) > so it would be quite easy to compare. Very happy to take patches that ma= ke > things faster! > > PowerPC doesn=E2=80=99t have an equivalent of rep movsb, so the PowerPC v= ersion > doesn=E2=80=99t have an equivalent codepath. > > Each tuned version is a dozen lines of code (plus a lot of comments to > explain *why* it does things a particular way), so it=E2=80=99s very easy= to add > different versions with different tuning. > >> I see the code takes some care to align the target buffer. That's >> good, but not necessary on CPUs with FSRM. > > It doesn=E2=80=99t hurt though, on the sizes where it=E2=80=99s actually = beneficial to use > the rep sequence the cost of alignment is negligible. > >> All that said, I have hard time believing the impact of bounds >> checking on short copies is around 20% or so. The code to do it looks >> super slow (not that I know to do better). > > The last time I looked at the disassembly, the code for the bounds check > touched three cache lines and has two branches. It fits well in a > superscalar pipeline so adds a handful of cycles. This is basically in t= he > noise for large copies but is double-digit percentage overhead for small > ones. > >> People like to claim short sizes are inlined by the compiler, but >> that's only true if the size is known at compilation time, which it >> often is not. Then they claim they are rare. > > I agree. > >> To illustrate why that's bogus, here is clang 15 compiling vfs_cache.c: >> value ------------- Distribution ------------- count >> -1 | 0 >> 0 |@ 19758 >> 1 |@@@@@@@@ 218420 >> 2 |@@ 67670 >> 4 |@@@@ 103914 >> 8 |@@@@@@@@@@@ 301157 >> 16 |@@@@@@@@@@ 293812 >> 32 |@@ 57954 >> 64 |@ 23818 >> 128 | 13344 >> 256 |@ 18507 >> 512 | 6342 >> 1024 | 1710 >> 2048 | 627 >> 4096 | 398 >> 8192 | 34 >> 16384 | 10 >> 32768 | 6 >> 65536 | 7 >> 131072 | 4 >> 262144 | 1 >> 524288 | 1 >> 1048576 | 0 >> >> obtained with this bad boy: >> dtrace -n 'pid$target::memcpy:entry { @ =3D quantize(arg2); }' -c "cc >> -target x86_64-unknown-freebsd14.0 >> --sysroot=3D/usr/obj/usr/src/amd64.amd64/tmp >> -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe >> -fno-strict-aliasing -g -nostdinc -I. -I/usr/src/sys >> -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt >> -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h >> -fno-common -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer >> -MD -MF.depend.vfs_cache.o -MTvfs_cache.o >> -fdebug-prefix-map=3D./machine=3D/usr/src/sys/amd64/include >> -fdebug-prefix-map=3D./x86=3D/usr/src/sys/x86/include >> -fdebug-prefix-map=3D./i386=3D/usr/src/sys/i386/include -mcmodel=3Dkerne= l >> -mno-red-zone -mno-mmx -mno-sse -msoft-float >> -fno-asynchronous-unwind-tables -ffreestanding -fwrapv >> -fstack-protector -Wall -Wnested-externs -Wstrict-prototypes >> -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef >> -Wno-pointer-sign -D__printf__=3D__freebsd_kprintf__ >> -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas >> -Wno-error=3Dtautological-compare -Wno-error=3Dempty-body >> -Wno-error=3Dparentheses-equality -Wno-error=3Dunused-function >> -Wno-error=3Dpointer-sign -Wno-error=3Dshift-negative-value >> -Wno-address-of-packed-member -Wno-error=3Darray-parameter >> -Wno-error=3Ddeprecated-non-prototype -Wno-error=3Dstrict-prototypes >> -Wno-error=3Dunused-but-set-variable -Wno-format-zero-length -mno-aes >> -mno-avx -std=3Diso9899:1999 -Werror /usr/src/sys/kern/vfs_cache.c=E2= =80=9D > > That=E2=80=99s a really helpful datapoint, thanks! That distribution matc= hes roughly > what we=E2=80=99ve seen - the number of zero-byte memcpy calls surprised = me the > first time I saw it, but the automemcpy paper reported something similar. > > If you can also extract the alignments from these then it would be fairly > easy to modify the benchmark program to use those distributions. > > My original version extended the FreeBSD assembly memcpy with a call to a > function that did the bounds checks, but that had *much* worse performanc= e. > We could insert assembly to do the bounds checks, but that=E2=80=99s hard= to update > if we change the metadata layout in malloc. > > We could modify the ifunc resolvers to use the current memcpy if you don= =E2=80=99t > care about bounds checks. I did a tiny amount of testing (mostly clang a= nd > lld) in this configuration and it was slower than using the snmalloc memc= py, > but I didn=E2=80=99t do enough benchmarking to be confident of that. Hen= ce the > CFT. > > David > > --=20 Mateusz Guzik From nobody Thu Feb 9 23:09:45 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCXdF0VhWz3pFjS for ; Thu, 9 Feb 2023 23:09:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCXdD3R5Bz3Fsm; Thu, 9 Feb 2023 23:09:52 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 319N9jsb010698 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 10 Feb 2023 01:09:49 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 319N9jsb010698 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 319N9j8h010697; Fri, 10 Feb 2023 01:09:45 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 10 Feb 2023 01:09:45 +0200 From: Konstantin Belousov To: Mateusz Guzik Cc: David Chisnall , freebsd-hackers Subject: Re: CFT: snmalloc as libc malloc Message-ID: References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4PCXdD3R5Bz3Fsm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Feb 09, 2023 at 09:53:34PM +0100, Mateusz Guzik wrote: > So, as someone who worked on memcpy previously, I note the variant > currently implemented in libc is pessimal for sizes > 16 bytes because > it does not use SIMD. I do have plans to rectify that, but ENOTIME. Note that you need two kinds of micro-benchmarks for this: - normal microbenchmark which does the SIMD-enabled memcpy() in a loop - a microbenchmark which ensures that the SIMD register file ownership is re-taken on each iteration (or close to it). I am sure that the results from #2 would be astonishing and give quite different prospective on the use of SIMD for basic libc services. From nobody Fri Feb 10 16:23:50 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCzZL2dJNz3pFCL for ; Fri, 10 Feb 2023 16:23:54 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCzZK2SgFz4Y4B for ; Fri, 10 Feb 2023 16:23:53 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=LZJdbhd+; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::f33 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org; dmarc=none Received: by mail-qv1-xf33.google.com with SMTP id i12so3858877qvs.2 for ; Fri, 10 Feb 2023 08:23:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=0IR3mSl+iVqnlcjdq7/lFp8WbcOF2ahpK206R/HH+hc=; b=LZJdbhd+dobdMvM3Cez8OQc65AEwrgT221fnzdd/udTR9bF7+UbpdcSZiWIGx4Dx3l dVk1+hYy6/sME4AOuP9HXj0INt32AtSbKUNTEnwejvbfSZNvj0m5LhofMba7ePo5tA7V zarsJjE4+zbtK/KfnRMWNoCW8l7hlswQKb1yjk/zd+u+z9jNCTdLo2BK72dkb7sK+xdO WHV+8qWNydXeIYvNpWBK0QGwmZgfZg3jyFyBBRkDl27hB9xFWa9jwWTnodQoNqZx7qHp 0zZpiHUx/dxitnqGYdKbiZJGScoOlQ1uC5WiN2C0A/Lhn3+VM/5PBZjdsf9B92JjnmNF 3ogg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=0IR3mSl+iVqnlcjdq7/lFp8WbcOF2ahpK206R/HH+hc=; b=WsSnF8H0aWO4d6XmI4nDuh/gf+D/zLbEaOqeN8UUGVUdt8He601NXrGMdZppNFNT+a XEUdtyOmh3hmbSIqaGIZqL3KxX3usShPz82/Rruv/Cnp9G/AjIf6u7JjFJQYKhqsyQ4m EgKucG297lyVlQIBrYcZQu9uN9cycKBLLVxPArk0/gnZ2LrZK4XOaDRlUerMpg8GhVyL tFaALBc8TbOFhrqmekrMXxLKyNU4pcUSryrlx9x7mx/g3AXohsTl6HS/pHRCNgNalUEM tGuNA5OaikIUZc+LH3SFqTDULLMgvsljP9+6TqSfNI92govU99aicFc6V88BFj84f4lX c/7Q== X-Gm-Message-State: AO0yUKUQE5Q/wam7Ao0AIshB3MhG4byMAZXETZeJc4UUSaP366r4naiF Yzcsp/AKb3ldadvu/Wa9IiyNHA== X-Google-Smtp-Source: AK7set9B5r2YI47y2zg8kjd1T4zrWsbPpJWEHGk7P20A/59VxjvkjdfsuIEKnTtwrY2ZqefGuEdplg== X-Received: by 2002:ad4:5bcc:0:b0:56b:f6a0:2333 with SMTP id t12-20020ad45bcc000000b0056bf6a02333mr25462176qvt.28.1676046232011; Fri, 10 Feb 2023 08:23:52 -0800 (PST) Received: from mutt-hbsd (pool-100-16-219-215.bltmmd.fios.verizon.net. [100.16.219.215]) by smtp.gmail.com with ESMTPSA id k189-20020a3788c6000000b0071dc769d5e7sm3772193qkd.56.2023.02.10.08.23.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Feb 2023 08:23:51 -0800 (PST) Date: Fri, 10 Feb 2023 11:23:50 -0500 From: Shawn Webb To: David Chisnall Cc: freebsd-hackers Subject: Re: CFT: snmalloc as libc malloc Message-ID: <20230210162350.gg5g7dihp3zef3ov@mutt-hbsd> X-Operating-System: FreeBSD mutt-hbsd 14.0-CURRENT-HBSD FreeBSD 14.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="7ww2svwck3f2m2k5" Content-Disposition: inline In-Reply-To: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> X-Spamd-Result: default: False [-5.10 / 15.00]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f33:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[hardenedbsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4PCzZK2SgFz4Y4B X-Spamd-Bar: ----- X-ThisMailContainsUnwantedMimeParts: N --7ww2svwck3f2m2k5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 09, 2023 at 12:08:49PM +0000, David Chisnall wrote: > Hi, >=20 > For the few yearsI've been running locally with snmalloc as the malloc in > libc. Eventually I'd like to propose this for upstreaming but it needs s= ome > wider testing first. >=20 > For those unfamiliar with snmalloc (https://github.com/microsoft/snmalloc= ), > it is an allocator (or, rather, a toolkit for building allocators) from my > team at Microsoft Research designed for both performance and security. A > few highlights: >=20 > - Snmalloc uses a message-passing design, which makes allocating on one > thread and freeing on another cheap. > - Very fast allocation performance > - Randomisation of relative locations of allocations > - Most metadata is stored out-of-band > - In-band metadata uses some lighweight encryption to protect against > corruption. > - Support for CHERI. >=20 > In the (limited!) testing that I've done, it outperforms jemalloc and > results in a smaller libc binary. >=20 > I've also previously managed to use it in the kernel, though that code > hasn't been tested in a while (last used with FreeBSD 11): >=20 > https://github.com/microsoft/snmalloc/blob/main/src/snmalloc/pal/pal_free= bsd_kernel.h >=20 > It is also used in the Verona process sandboxing work, which makes it easy > to isolate a library in a capsicum Sandbox: >=20 > https://github.com/microsoft/verona/tree/master/experiments/process_sandb= ox >=20 > We test on FreeBSD in CI upstream and the code is actively maintained. > We have implemented compatibility wrappers for all of the jemalloc > non-standard APIs that FreeBSD's libc exposes. >=20 > In particular, snmalloc is designed to make it very cheap to find the sta= rt > and end of an allocation, given a heap pointer. This means that we can > insert bounds checks in critical libc functions to prevent heap overflow. > This is done in the branch for memcpy, which some investigation of a corp= us > of security vulnerabilities showed was the root cause of about 10% of > arbitrary-code-execution vulnerabilities. >=20 > The bounds checks are controlled via an environment variable > LIBC_BOUNDS_CHECKS. Setting this to 0 disables checks, to 1 checks on > destination arguments, and to 2 checks sources and destinations. An ifunc > resolver selects the correct memcpy implementation at load time. >=20 > I did have a version that checked a bunch of other libc functions (e.g. > sprintf, puts) but it was quite hacky (and the way the ifunc resolves was > implemented broke tcl). >=20 > The current branch puts two things behind the MALLOC_PRODUCTION toggle: >=20 > - The additional security checks that detect corruption of malloc state. > - Pretty-printing errors. >=20 > We are currently separating the former into separate knobs upstream, some > subset should probably be turned on by default in production. The latter > has less of a performance impact than it had and will probably be on for = all > configurations at some point once we've refactored slightly to ensure the > compiler can tail call the failure function (which moves it entirely off = the > fast path). With this enabled, you get errors that look like this: >=20 > Fatal Error! > memcpy with source out of bounds of heap allocation: > range [0x14823c02440, 0x14823c0246a) > allocation [0x14823c02440, 0x14823c02450) > range goes beyond allocation by 0x1a bytes >=20 > Abort trap (core dumped) >=20 > Without it, you just get an illegal instruction trap. >=20 > There are a few limitations in the current branch: >=20 > - The memcpy integration is broken on non-amd64 platforms (patches welco= me > from people who can test these!). > - Only memcpy (not, for example, memmove) has bounds checks. > - The memcpy in rtld is naive, which may impact performance. > - MALLOC_PRODUCTION conflates too many things >=20 > The branch is here: >=20 > https://github.com/davidchisnall/freebsd-src/tree/snmalloc2 >=20 > It adds snmalloc as a submodule in contrib. FreeBSD is allergic to > submodules, so upstreaming will need to replace this with something more > complicated. You should be able to cherry-pick the top commit on any > vaguely-recent -CURRENT. So I took a little bit of a different approach, which should provide the same end result as your submodule approach. Note that I'm doing this in HardenedBSD 14-CURRENT (the hardened/current/master branch). 1. git cherry-pick -xs a5c83c69817d03943b8be982dd815c7e263d1a83 2. git rm -f .gitmodules contrib/snmalloc 3. git commit 4. git subtree add -P contrib/snmalloc \ git@github.com:microsoft/snmalloc.git main I believe this should leave me with a tree that populates contrib/snmalloc and pulls in your non-contrib/ changes, leading me to end up in the same end state as your submodule approach. I am seeing some build errors. I've uploaded a WITHOUT_CLEAN=3Dyes log to: https://hardenedbsd.org/~shawn/2023-02-10_snmalloc-01.log.txt Note that this is with llvm 15.0.7 that just landed in FreeBSD main. Any non-XKCD[138]-conforming pointers would be appreciated. ;-) Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --7ww2svwck3f2m2k5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmPmb48ACgkQ/y5nonf4 4foIoQ//Y90bPyFfbRmK8vPOs0cZ4EKWuui5YZvyvv1BxM7yU9VFWeVwF/C1HIP1 0MKkxRyxE3s68K/xMBj2VCx6OgBShofVlzIkz9RWiza39CliU5Qx0ZcSZbDRT5ng UXTCut/VTIbaNJtpZMYswl6tdMs4U1yroi8r9UsytmGkV9PREl51YDFTPnopVmSC m79/INU4IOdz0uee9LDZfx95G1fU/sgjjNL9dQtTHuDSrq5iOynILlfr1bcEoR1O IZAOp+P36AeqodHMDh+E7G/vO04qhR7MugHkNpUP/WQEIa0dLtffo58BWnvzCymd h9I8wtTl5RGyy0gkgnrQWdOEEz8GV6hIMDtr5/LrL/YHQUPC2xoF5cyvIh/4fmLr oxKY/GfCDu49wy5sHZ14iBM5v8zGESZK1nx19XX/vctpwV3xQkH6FnOqGFzX9Rkw vLR5AtGOSf8Wsk+4Doki3UjG0Ax+KzCWhQGkKEW2hNWHEz9k1oq0gBykURSedqEA UPqabFF/bc7TP3d6lec8OwFmaCjCKZBmLZxvG4K3tDYb+wqtYZ6EeNDJ29gamFmW lwxdHfOuPE/NrWx1f/Ie9a6xkTTM4o10397pOBqeFmEWzvAkz7k928ib0INxh1qP CAcfH7NaDFCAI2Vfeq2NckdRXdYwOqFFDmlDOi4sYQUdam/8LMU= =8Zb7 -----END PGP SIGNATURE----- --7ww2svwck3f2m2k5-- From nobody Fri Feb 10 20:01:00 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PD4PC6xqkz3q00l for ; Fri, 10 Feb 2023 20:01:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PD4PB6Ys1z3m4N for ; Fri, 10 Feb 2023 20:01:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=r+lo4MTu; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676059276; bh=8yeTRRm4wlROTxpEpB9L+Fz5E1ezuXacsnCv+94GzZU=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=r+lo4MTucxt1uszxuvgcrRZy5Wp2DTmOVo+fgOXSOLkJDp13oDdRWeb/TcZSvUoGqFWiriTvWcfR6z8H4KAIjtDluTl8KtzvC3OCikANACy3t4FqHKpnyzigbueYWkO67EzsoCxKwMZWG86VwogR7S60czrr1IeMzFQrpsYZuBT5pPMXVHTAgzipVEwl0nhfOTREXBdqBT2KlxOA89M40S2NmjBIyDQyfz60dU76lKpKILCVlR4R5/iu6c+dWax7XFifaNOZ50F1hb55zdjVxzusgFv+JctimBOlMDZ0sJqq77FxL/qFcx4VOBN4Hww0lpwNiS4IRYMvV/O3NniRdw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676059276; bh=3Bxmaqh690N5I6tW3sslXm/G1JZmJUUwjgHma2vwBJN=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=lqY763xGr9eQkPt59VFdEWVhnJBxYIC3Rs9XjUP/gRSFYXwYMGpt2xzKhYFUjMawqaE/MpxJJZG7UiAVdFdFVDDUOhB9PvAln6r13pAI+fD4+xDV2SEMYTaSIwfWBOhGRi3KNfmyl/fDilZykAAnk+IZZwHV5RPX+y6oT9DIbT4rKsEWN1AYznoXCxVpDdrUr3t8O2I6QJkDRMg1Oz3h4nrAyK2ugIFLSIe/eiDKMldbPuAaZmsd+PsY/1pi6LiYGITTtsPIXDISFNI7CL7Ifu0Y1LWYJYYtj6gXqoUaqYtm3a+hAYJFOYzgodbR7pj3nG9au5fxm1npUBuIWiHoJQ== X-YMail-OSG: NsnjACAVM1mrjnvw_3pD4Z2m9lACj6P4nFjHcWHt.dhBZF27jbcwpoyDmy8KPI5 TlhdmeXjajq8E7MXIQrQrAEMYe3CNUgZRS892niHBMtAor6_RrXohTc6IbNYwIxBErazfhZcjiyu 37juptMGmQYO84ykzOpCuDiHwc.gnLu6o.uoecrYrt7r2jU.Y4AjzqysdymJmSNWPwnvx3xxta1c n_qbbsuWZ1cFxi4zPJDRAELxJ31AhKGdHTashdvStBW99YdF5Bre1VAyn.mzq_Vyg4QlworE7QNh tRxFKrLHrWLseO0Tj8a7QC_gI2d0yn637zatWVxjoympypXLFnWuiCjnhkI6lhBoE5B2vHg9KKGW q4I9IloWcJyNIgikNtLBaD3jA2NY7_oOxAb3yEKVzej5RECNCTLaHF2RgVL1BlQO6puGfEFRpQKt Jh.dV4UxtzSVbizfV9tpXomAgmpE0L5Ky4XSxWWb_YUp4Py24FTUHBO93AF6EbSSQNe0H1FWTL6g pLn4h4fCnI44A8FubclMerHwf7HBnn7qwuahj9dNwDWOFRJBq8LY_bm.a1cF5p2C9kPzEyuB10yW tt6fviBGxI9wGIbNeErBvBfaxhHTWYnMet7oO3oG6qYtxT..f_egLKXriRzse1hy_TR0qLugRInk RWMPv8CMLpQ1GL.bIIXQq5BCX33rI87Hw.i1YVVlS4hZypXGipLaM.t2TI1BeLBZikwZp2kim7ng SNN9wOqEAR9IcemTrFBfD1jUJ0k1pmNSuagVTgjvWwBiROp8khj.WoZ9bgFwVWNzCgRVLYdeqCqS gtm3ndl4LIRNQIgf5SuPmdJJ7q29mk8TWKwYFEKQUtJPcj0Vk3ClAmquL5F7I.6gBptW84xrOpX7 AdOFCSi4LsFTOMXV_TD0Jmki9NboKVLMRri_wKYrYURF9twISs7mgmeD5eqoogYTm.V.D9jpoJaJ d2bis35_jDUBKnWDEWoogdYo9xXP40iYsn0hojeANWn5jl8wVwpIQKliZYbL42GHGdycOeGZU68o HqXn0p0JcTgRvu7XCoMwDupPdMhduXMGJZvjEVHx3v4GAht3h9K0Y_hR1PRooD48o8huS6ROJExZ L0RlZRpM_3vWIjMiXQa9aF6D6ytzJROBCf4PSjVlpH6Rj87OoZRT2LtWFGiZjF4Mel2XYVPlW2sU tTaaqV0y56BPk7e8Yh9P2s7CMGWYR7cs_TG4X727DEkr2uPOYTmoc6MBWUcyQcyUcw0cUFeDSC1b XvSOJtj8GIGMV9Be2vx1aMrJZ7MC.1.kRX4idw9q4HVOi7WyAxq.7bvZjB.6pKcVm7qMgyc.47a9 LiJdG0rR8z2CGs2EsyLDlQ5xbltnuKEJmJexG5ocyqxSVWsFVXTV_MVr.yQ4LnKrqlSEWvPXk3e2 _3SdOUhF_YS4AZAQIlL3qE2W..hXDgDlwQ8r_bpbvRtdiO_vRNoDYgzlMK4Xqwl5wgj1h2fLWtmu x4y5v845nOZSX882k2ekYvPUk_gU1EHGW6ER.q314lifm.KSFeOBdxI0jEbPCg1Csr15.KZEt2lk R6MLiH7vUarCdAZlYQLq9R4Uwtecyx.ZuYZhWEJky8VWytL5h380D7SbLT3AspDQl59aiuLhPVX7 .wuBowCBrCq6RwakvIwp4nH69BjJdW_mb1WU8vdpTyR8.Y6OCzeqXEj3xiP6.oBSL6z.OkkFVgI7 VROrDVkW26fR2CfJSZTzpyu3t2J2LEkc2t9lSbt7mI5mBQrPqMH7z9ssT.CuSq.dHH365MsXetma HKM1T.uZiSd58zteE.njHHgqNexxstO_NA2p7K28T6TzLN5i.j66ltKWUXhUJOfum.cnoLYb0IRY 07XPRVXpF6KCtFrYZg3uCdWqdkMiih9Xg84U3Nc31DgocgiLetOOEesl9p_Swdvb88TVBfQjKU7W TioZ0eer2hcVuBfc7k4lvKO4hBnPscAVtqbt4lBdgWls6iJjFdzgq3gyrFuzB_pFcpPmVhTOeByj GGUNiIBgSFiEsW6eHpVzRfdCJ5IJl4idKy00TRz7Po6PHgGsDS1Llfj702WvoUyaI8x4jedCMC2n bx7RQYKt2MryJZs8DyjZ9asi.lkpP9ZENMT7wd4O2YGcaQTOZquK.S3wFf9YwtRWAQFuddLTAKVF _8pQ82TwErAe_fuciuFyZhocvRHvk3XA5L4wMur1jEB9_caZ1PddBjpZ7i_2wVdOltOVTYqH7U2j WKQIwiKWfxxNtz_yvypSzo1.hrLjFjsVGBxRSSYF72Edkxe3pUXmCr1oa10w78vYqY7SognMHBno SAOAtn5_z X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Fri, 10 Feb 2023 20:01:16 +0000 Received: by hermes--production-bf1-57c96c66f6-npzd5 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID af404e276356c9cf8fdc3c464d1f634c; Fri, 10 Feb 2023 20:01:12 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: CFT: snmalloc as libc malloc (the nullptr_t issue) Message-Id: <22F8AD1A-C1F9-48FA-BCED-E165E2DA5260@yahoo.com> Date: Fri, 10 Feb 2023 12:01:00 -0800 Cc: David Chisnall To: shawn.webb@hardenedbsd.org, FreeBSD Hackers , Dimitry Andric X-Mailer: Apple Mail (2.3731.300.101.1.3) References: <22F8AD1A-C1F9-48FA-BCED-E165E2DA5260.ref@yahoo.com> X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from] X-Rspamd-Queue-Id: 4PD4PB6Ys1z3m4N X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Looks to me like FreeBSD's problem: there is a rule about inclusion leading to a definition of nullptr_t that is not being followed. The details follow. The complaint: = /usr/obj/data/src/hardenedbsd/amd64.amd64/tmp/usr/include/c++/v1/cstddef:5= 0:9: error: no member named 'nullptr_t' in the global namespace using ::nullptr_t; is reported against text in FreeBSD's usr/include/c++/v1/cstddef (so against the llvm15 integration). cppreference.com reports for nullptr_t : . . . Defined in header using nullptr_t =3D decltype(nullptr); Notes nullptr_t is available in the global namespace when is = included, even if it is not a part of C99~C17 (referenced by = C++11~C++20). nullptr_t is also a part of C since C23. . . . c++/v1/cstddef has, in part: . . . #include #include #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER) # pragma GCC system_header #endif _LIBCPP_BEGIN_NAMESPACE_STD using ::nullptr_t; . . . But, in FreeBSD, directly and indirectly does not lead to a nullptr_t definition as far as I could find. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Fri Feb 10 20:53:18 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PD5YX0jxRz3q54D for ; Fri, 10 Feb 2023 20:53:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PD5YW1pFgz3rbP for ; Fri, 10 Feb 2023 20:53:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=maEmdPKa; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676062413; bh=biworBqv1HrT76ez1LBWVtQJgo+VySKUr2ipI5TQp98=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=maEmdPKahkhtrq4CqNcYr/qugtlbmytVNp+TkF5krUga6df/xmwG0UHhjCALO4MBiDAI6bEhlYXkaAIX40l1vQ+2qzJ+FVZePHdeDatFnJXoS9DSTXvO63ZNpOxzO4wqSg20UIO/IgDFySQqi/+w450AMzzVdWj04iO+pCcQdNbVsaCACcEotS1C3Jpycx3e6WQV3gUUmzrvTDWHUuhqUpWoGT2trG2spgIX8Bw3gwkCJTZ71QS1PuI+D3TAkhEv3z09rD7VS4yWB/wZqPFzHKbqJoWBeVsXEYBdnChrczwttBolNHEQBf8ySq1ojWaBr6XMDdqPIGRCqUxYswhYdQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676062413; bh=Sr+cgbyYlHHSmvfdXWapymcnrziO4ac07kRKBdIMs/p=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=Dd8ZJ+bsUa5r0MJCAR9ls3qocGxRkm5IqEbCkL8oFfvmS32rYIsuGaj46Ya+g71R0JmvKzDmN7E8I26hbrlFKs8n2OIaM8KoLB5XB448CHSvjVkPK2lV4FOUjbqDjZ2JBlNvnlk4EXiVodfHzV+5wcmRqXQNG6QnrdG6CZIOAhLd3ZasGnGeh70vBNPdvW+i9jL1Os81HdphTWTMw9za9b3Al1yxK3D413gUXGpm35TAuoynVn+Qd3wQvNN1qWDKNBu7ufTa1moa+x5nuQkrwRexHCaPtdE2IhYL2lWmc8dh/iz7pRkzy7KrYZ2f+Gb/UKu8jmN1PXw0HqLOw6nwSA== X-YMail-OSG: NtbHtcEVM1mAlrO5BlDRlTsmOPq7g2RF37_xb_qwFbBjko1BmFLXKFJX5u3rKig K_pZoAE6z4AVj5pT.b8xhKPkhoGs2uD6sTWeVQeCMNtHjzT6h1VAzK20613TDVZ5BqH1NiIjPn4Q LLNR6N0d4fQGI4eIeYzdTnODcHjA8t_GDj1n7QTqh029saekUZSKlRhBzEguypU_.KhJSuV_jLM1 EmVnJLnTjFQOOZDJd96LKyQSxhwWlIUikTSfuhlUnjN5U4zuN6ETHFgaYsQnFclcJBPPuJvuj_mG EBICqHJ8sflikxtKv_W4rn1gxu5o_m_qCsJvhyPevZyDs1tOPl_p9yLWnkMpybE6H4OkdVQyFdeU xGFWHJ8DDE9hmfYbKq5u1fL6KgD52SnmIsTt1nWMccKnwGs944hFULh7mUhtbGoVWCX.yYH825mZ Hvn2lgDPzzDb2YEIHEXyHTMdD80Bdn2Wy8kCHSafpo604WsmKDdbKgkCFgTErTOxDOTpg6HT8i2z JhEUgISfFXeYx5NV.Zjo.np5WjbXEhWS4XuuOs1FALTmc6aZIwYNH70cwVjIRBOpOjCqIIpoTPwc m8gGiZpj4OwMvWZUfGAmQQfVQEHqAjrKVV78mw9UWWmyjJVPpwLby.VXivcM_xFjFsadrrKUE8eO p8v49ZtUpxP.mpt.S1fKJfplQSBinOA1cJ.QADX2_lVtgTGvMihsMmxxfesijyDXi1n7pa0FZVqE tYoc30uZO9QXSWIeiGIrPFeKOeN.V5YaQY5Z.JVaQZxr95bADvG.WpL33mNaYh4TElo.wheAJozt .XOQ2xczuMF534BaJrw0iN03gYSyXePNv155.WFfwA.mgcKOTF0ZiTQxdk2CEnuX1wyXTw8qwF8J YTwuAW0vqxI73_pUK.U9AMpSeW1nqSQH9nlamRDjemYNOfVZVqYb42gqpDZ_hnbWxjy6UGR7T7dG xgQTgQRsBG5azSnvMrBopmJR2QAHxbffLKKVGpFOAscwg3VaO6up8YdxApyp.o9D3OoYU2qbwc5z EBqQLP61wnzxarNTWpnio2Smgrt43z3wMcYFPk4kyHX_4dur5xvAc7FfNfaVYyhQczTCdtVmIcOg T45_F1cHWPlJG5J3E_run1JMcbs77_veTE_b31o_WBlYXefsPUvapXj_YFIBwQkxjH7ouhJjhDyl rzSfxPuZu.Hbba6Cotr4a7hFRF8YvAboLOjJ_2FyGAL7jxjusxxrE50l8_uPXCIF.h7zxHE3WRFW 0dxAvfoUdb9qSxEPWKX65Di49rV0ZGK3jm_yNtMTrLcRSvavskRJOuVfM4BXaabEKZlNJ8PRFaIE pmO6WzenYLO.6BklpPQ045S2UyAFR8AjlG2Akw2kHdO90MwQdaRUV.EznRqY6LXKPu39pQPkghoe QO.5.EnrESD55gKLJASgiOXdwRtmvXA3no6z3KQp0gR9I4ZzMsHyaoonHs2qBkTOjRRjQ5XeHfQz 7yD2Ia78pw62ivUrQKWzU6QzKsHJQ3xF9dh3peNipJRqKIhPGpGWvsRoluDZB1Wbi7_jzMsbGqoQ gBFAZskyXbJQijvWM4feYn2SSO9kBgJ_rPmsHZQVPLPpOBavn.6gbbx8kC5kjJSLNEQF3aE1Wyah 2ENxDRRqh_AWB.uzP3maHqLWuX2R6yGMzFoAZjTGiRvIwqyRW4gHj8p1kAlRRvEfZnMdiTP2W_ka sJHVDn7XLdeAFOrDTQzwWkVDUWhtIQSpWqT6itnUekcriz5tU3u6dmWjBRcLjFfSU9_tdSyMlHgA 9lBGbTuPHj9JlrUrDy0cPFGW.WxjtiBcT6.RJrK.kCvjB0S_WdsqMRSzLko39xYhcRKih8_PrJKY 4sDu4QO8unF2FXw_c8Glrzm3mQwl4WfePfjQ2El5UbLelKbXDyuzYj5ledkihMWeyFKx.ZjgMHGB GtfAZMN83Uq8sYMugNiUcofslvzv6wgrz5evyu786QJl9gbB3wiQQLfhCRpPog2fGqLNt8C0oBsB OS2X4eZHWRwWPHTfugHo7dMY2Cc.hM6UAn3qY3C32.EEXNi_gqx8xOPPMtkAhSC9aRnl6B8NuD9n .si_eEvEfoHvvYaWCZuUbyrrLFOm_CwdhBRTlf9B47OAVOuVtFuxYRVCaY4X5ITREQYRibC1toCH cBg4ZYdjCIVgX1eQZYhUQPKQCK1L8XYf4BXPdieruGzuHemkNR_QJ2K1R5WyONqr8F6XkHNbkT2D R0yfF9eosAvxwC4OoaIvYbjKaDcN.rUGLvOSkuoCdAIEwDzs21Nj66.6w0pJmX5pjGj6J61pUMco biRt6pA-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Fri, 10 Feb 2023 20:53:33 +0000 Received: by hermes--production-gq1-655ddccc9-5z4vz (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7395b48d138e4cddcc536081fc5fc359; Fri, 10 Feb 2023 20:53:29 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: CFT: snmalloc as libc malloc (the nullptr_t issue) From: Mark Millard In-Reply-To: <22F8AD1A-C1F9-48FA-BCED-E165E2DA5260@yahoo.com> Date: Fri, 10 Feb 2023 12:53:18 -0800 Cc: David Chisnall Content-Transfer-Encoding: quoted-printable Message-Id: <1982263E-57E0-4709-AAB3-DBB4F75C7546@yahoo.com> References: <22F8AD1A-C1F9-48FA-BCED-E165E2DA5260@yahoo.com> To: shawn.webb@hardenedbsd.org, FreeBSD Hackers , Dimitry Andric X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from] X-Rspamd-Queue-Id: 4PD5YW1pFgz3rbP X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Feb 10, 2023, at 12:01, Mark Millard wrote: > Looks to me like FreeBSD's problem: there is a rule about > inclusion leading to a definition of nullptr_t that is not being > followed. The details follow. >=20 > The complaint: >=20 > = /usr/obj/data/src/hardenedbsd/amd64.amd64/tmp/usr/include/c++/v1/cstddef:5= 0:9: error: no member named 'nullptr_t' in the global namespace > using ::nullptr_t; >=20 > is reported against text in FreeBSD's usr/include/c++/v1/cstddef > (so against the llvm15 integration). >=20 > cppreference.com reports for nullptr_t : >=20 > . . . > Defined in header > using nullptr_t =3D decltype(nullptr); >=20 > Notes > nullptr_t is available in the global namespace when is = included, even if it is not a part of C99~C17 (referenced by = C++11~C++20). > nullptr_t is also a part of C since C23. > . . . >=20 > c++/v1/cstddef has, in part: >=20 > . . . > #include > #include >=20 > #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER) > # pragma GCC system_header > #endif >=20 > _LIBCPP_BEGIN_NAMESPACE_STD >=20 > using ::nullptr_t; > . . . >=20 > But, in FreeBSD, directly and indirectly does not lead to > a nullptr_t definition as far as I could find. >=20 I missed the fact that there is another stddef.h: /usr/include/c++/v1/stddef.h that looks like: // -*- C++ -*- = //=3D=3D=3D---------------------------------------------------------------= -------=3D=3D=3D// // // Part of the LLVM Project, under the Apache License v2.0 with LLVM = Exceptions. // See https://llvm.org/LICENSE.txt for license information. // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception // = //=3D=3D=3D---------------------------------------------------------------= -------=3D=3D=3D// #if defined(__need_ptrdiff_t) || defined(__need_size_t) || \ defined(__need_wchar_t) || defined(__need_NULL) || = defined(__need_wint_t) #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER) # pragma GCC system_header #endif #include_next #elif !defined(_LIBCPP_STDDEF_H) #define _LIBCPP_STDDEF_H /* stddef.h synopsis Macros: offsetof(type,member-designator) NULL Types: ptrdiff_t size_t max_align_t // C++11 nullptr_t */ #include <__config> #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER) # pragma GCC system_header #endif #include_next #ifdef __cplusplus typedef decltype(nullptr) nullptr_t; #endif #endif // _LIBCPP_STDDEF_H So, another way of saying things is: this one seems to not be in use but should be. Note the dependency on defined(__need_NULL) for the initial #if . =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Fri Feb 10 21:34:07 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PD6Sc4DSbz3mwhy for ; Fri, 10 Feb 2023 21:34:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PD6Sb3hGTz3xj4 for ; Fri, 10 Feb 2023 21:34:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=pwpXgC5p; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676064861; bh=/K6N17xw9yPvc/Ld8J93p4uCuhr7eQdJdqQffI1rOF0=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=pwpXgC5p2P7rIbefqGe/pNIyWIksyaXIya8L13J1316I5fpp75kJ16rtVf5hs1rBfJ6ePS/OUILOd+jPSsbmDD2n41e78HnQ4h9kpmFcQ6OipBu+f0dfdVAxE5C+PPoLaxeMonXR/n1OOS8Zik6OtL6adyxJ2pv0j8JyMBG9LAI6ncEX0pOH1EuJtnittsuoOGjjbzmgqhfp/F/LzAJLLilBvGHmc8bEU+PI5wCx8av+0WY0uoKTg/z1KL+NWFeYh8DVDoTs1nO+Ckylw2BoXkKgBL/NKgoJXsoQhQ+oLcCeZIh9K1xrGGlIFmmXTHoImoIwjJYFOoPvebHpbTXa4g== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676064861; bh=tXQoD8d2Lrmwy4pwcaxzSGTghlU1F/mdmqTlHP95ohu=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=k7n1FigAEHhInjNc0MXdar9YzSqLZjpl5LS37f707w1ABEGW21zXVrnR281qfDZkX9fhbTfoNyNhjIEBYszRucCND7dLdX8xoCU36HF2P6QV1lO/yHdVQU5mQvTs7jIDaRdTV88tITpm/GAcGye0qZmqWrdQq7LWQRWLSsfWQ9TpCi2HHxzLQvzVangOrDF4zhABFdmVXkTd1HIbEemDr85CCqBOiQwpdkARpuwiQ/u7vqkEXRQE8ozlL1idI/EmL5VsF/qWl00fKi1I95RVpQddzTy8Ruo4O4zFNsVEpiPu68831V1G0FibilBkhPoy922/skpKH9JBpOLQu7AfXA== X-YMail-OSG: mjDgmcAVM1m7qAjIJweyLwHFClKE2lLP9vBfX8EK3JWRlBi7_5JMJ8xRrayZmBc Z6zgwu9nixqRhr280N8_jQ8xherhX6ENtuFk8zOHJRWB1NFSl29OIbkaIgjaiWJmcuLisBBZwAsS X63L1w6PXlmrThnh6W3b5fUvhP1PAVdnT0FDUfOZT7yhh1zX2aYvIzu4ZNmzmSvTb_vAfhzoL6sk gIizBMMNW9MLmmXCKT_YGbQsm.fhB6HSVyPBDwrISsSCUfmmda5ZLB6pwl8ACxGYgt8buxSmAfI6 LWQ3GeOomVSMRIRtuENm6U4rfMDUUpxFYDCV6yl6ELEQNdVZI2ibkumsgDYrGXCO6QzkR8ETG5Xv Fyr5tjaHcsCjNsv1sc0HET5e3ekjlwDMEHsP0O.aoCm6Ripz0.dFkRanIle4_iM1nHn1lBzgLPXe owWBDL6jkgnLOHpg.g_zxhzJXUINu3N5i9_vQOnA2G.SPLN_1WxiQy_YyvHo1z05Oo5CJpp0lTny fiSUpMIKhrd2rTtXubPkLpdNTy1p4rIjYWTMGNGeiaNaDERX2gq8B9iRctFba.YZPD1aTdP9Sg5P 40Q.29IKx_lpaw2g.T4BIrPwjbR.qEtL48ZnmHKqTzcUzkRJLk8a4DAsCVEx2asVhjKat_Hd3OcH 5mAtmcSmT2dAEqIVAvA.M3F9sqUFkJYvbQI5OwNUgko7vBmq8f5fPZl6fbLBXPygUzUisyOdPWUO 7ogwPajZrqKfCnfV3Oopz52KrhVgx5c.yeuXHENrxtaifau45RDieEOhOtsehVjB5LrNhgZFr5n7 AV5rwjqw1bKJTzNamOWkgbRzoSiMi_vl3VLcXWGQRRvuGRVfj.W2WVQ5bck90m2wezvv7rgAL4Do R.M6lXRV5hfJprTxi7MrCFwTRrVJQfz_ygrzIElZ132LJs9anOF4WAZJZZvRlypzYtt24JjCPVeH 0G0SbA8gy3zHaaScIZ4gV6k4u.MuU59fPAAxGR.oiD47nJTlAxz36HVB.6Bc1jwtbGJ.GrUZTmz2 WwOc89u6iuwEJaWM_uOJSRp0ydTlGnDHvbAcBhK.zEtK2HSi.S4TSEJPR.A.TSuo_g00L.Zh6lu1 GUg.LjSefHAqNQ8ffo_VHF6A.OP6DFYm1SsdyUD6zfls34iuptDrBavmp7dUvojDTQsQnjz_7sNy 1muARuVvzVyd4cjK7QuidtwpTgXYzCgDeDGaCoLvMR7J4QQhcTxw50RsYJlGXaBSkt4EoDNOt3AE 3LihZpomL8ze_51jWmxXHgwxAZTSeEkI6OPk4Ar4osCCpQMrXnCLZ50XF4mhigNH1g5WfUCak8jI hBK5wEIlHoL2JCXTff_bvFfR9gsRckcjo3buuYrTFPkAhovblmNES5WMLhS0SuWx57sQhX2T6fGE f03FG8tI7LkckSN0M7_gf_se52lsRIleuutB8_YuI.eZccDY03lS9f2hoA8e.uD._P183EydUtd1 GL7Jjlx38aTdxIECl3pyxhNjWvC50X3ypTqnxuaJ33PT68hXuyrsKykhVGbTmQ3ybxwifi.Ww3IJ vGthHM9aQHyIohtzvsqonGiH4O4CphdT2iopEKoY9tY5hz.BQKdc3R3fd7jV_tVWg6yg8pyo.pD3 G3P0lVCVqKAVVU6eMAQYjTZ.xZKfXMqkW5qtyyzLRJEoGvSUnno3JsKG.MmL1W2zObHq03LXNYqs tWx7j.5yZSzOFTFEf18tQOdA3q._QdExS8OcsrK5.mh4bY7OVm3vGFG83vMBNF0U8aRmvaneDz7B D3l6AH.9UfjeDfbjrjLVDidU0ict5MYFcThm_iVi8cIZlmYJl32oXG7FYC_QDJb3OptI.fFHES7o m7RwoUf.pBJyQ8W5QmuAo4v8swDt3wrslgRW3rZfzkkN8SFUx4wgjvkR7E0Jmg7dZx40J16iGH54 ss32XvHAem9K20EBOBAijqL8Dr2foSFoHLdoxXe3P9K5VIcj4GxTBxpu.IDSLmpgvbuJSMjCbrC0 cIYF_wAemLuWGuUn33ACAK83wPc1qvlKJODViMmwFVVHAj0YfvnImKNCfWJ_x9MIJpL32ifqi_.V zzN0dY5Df3M8JJw1AE444m9yz1qycHlF30rn4TaujvovVEljEYesLNm1865kAtXb_EjlFWHq3GuQ wYh6GHV1ZjPskghqAweNS2yccqnO2fRyDpCVwiO7wcBoenYsqEFVGZ1GwcpGKi1olG2mSlxlo2Sd KLyjb7tB4eQMypQQlnENjpVx2GA5HwOtaSt99UTffYWwzAshk.5EnHG_lYR1jiu8Vg5OvKwYGxck mv2.DOa6s X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Fri, 10 Feb 2023 21:34:21 +0000 Received: by hermes--production-bf1-57c96c66f6-kqcsw (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1b8393ce7649742001d1bb515d039252; Fri, 10 Feb 2023 21:34:20 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: CFT: snmalloc as libc malloc (the source_location issue) From: Mark Millard In-Reply-To: <1982263E-57E0-4709-AAB3-DBB4F75C7546@yahoo.com> Date: Fri, 10 Feb 2023 13:34:07 -0800 Cc: shawn.webb@hardenedbsd.org, Dimitry Andric Content-Transfer-Encoding: quoted-printable Message-Id: <96652C51-EAB7-46C7-9039-557A5DCDAB75@yahoo.com> References: <22F8AD1A-C1F9-48FA-BCED-E165E2DA5260@yahoo.com> <1982263E-57E0-4709-AAB3-DBB4F75C7546@yahoo.com> To: David Chisnall , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.147:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PD6Sb3hGTz3xj4 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N # find / -name source_location -print | more = /usr/obj/DESTDIRs/main-amd64-chroot/usr/local/lib/gcc12/include/c++/experi= mental/source_location = /usr/obj/DESTDIRs/main-amd64-chroot/usr/local/lib/gcc12/include/c++/source= _location /usr/local/lib/gcc12/include/c++/experimental/source_location /usr/local/lib/gcc12/include/c++/source_location So, none for FreeBSD and its llvm15. This makes sense, https://libcxx.llvm.org/Status/Cxx20.html shows: P1208R6 LWG Adopt source_location for C++20 Cologne Complete 16.0 So, likely FreeBSD will not have this until it progresses to LLVM16 . It just changed to LLVM15 in main [so: FreeBSD 14]. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sat Feb 11 17:10:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PDcY95xvYz3pfjK for ; Sat, 11 Feb 2023 17:10:05 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PDcY95VdRz4HV4; Sat, 11 Feb 2023 17:10:05 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676135405; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p8Ll62jtVCdAD563FEDxQL+xOGDd4/78HLVokc2rbPQ=; b=MMjHincxNgq5JWsvbd6Z1hlJPvQSWclMP9IUFMuUyiUMzeMmzpxYtGGiELehWswOqz0Kj1 NYKWiKhe4H76QBRyDf5VfhX93EaKJljDKGigeZ+xoAtAJp5Gm3aELJaC068oOJbn7G6gub y4L/iNK8pFB1hmPPDxL5+GhtaQB+ZXHgeKWmrs1OhjSJDR7R8M4e6LHbwWG9vA62B2p2SL hmyjluc2REzF0ZTl7V0hZ5YM4M8kHSVT39eGVfvnonK4Kt6xPyMPIKvZEPT/qK8UU+4ys1 Kt6IBF+Tqhs/hCgl5Y8gZktAVmHKlfpvCJmL+LxEUcAVObwJqjHfBK9kt1hXLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676135405; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p8Ll62jtVCdAD563FEDxQL+xOGDd4/78HLVokc2rbPQ=; b=wTZl5iZ2dUFVD5SAldHu8SwUc/xlxPj2QdA/R+/iPP/0pMe/xqyJD79hoRxQJZmOnhDe/x 54K1nJFCvZlCzHm9A6ELxUmElr1SUlrfQSxlodNUj5LvnwEAd5FdVZrGwLWJsDF0RvCfz8 Ul0/OlENVMZkBEM+M36N6JoI3/+tEuQveE3+6BHzmiF1t8igx45V6khNSPccy5QHtimOpo ZkfoqTQG/XbWCVMr/gTyhJjqN+HHb5JujrRFaHzlFakvjUqNAJq1f92e3gDDA76d/BxGlI 7N3kfvUQjlqkKnduYfwJDY2p3gvJQn9OcHj0e72AEvYu0b4vdAM+B9sUi3AyeA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1676135405; a=rsa-sha256; cv=none; b=HRdaGFc9b6i03UdpRXHVlnBU9HKPa+61f+LqElS2KutxCLvDrPTFpQsJzRGm9WYFnPdfUj F1HCgNES8FL9txaFA/dmVE6XuFAI3jNcjuVX9b/P54qj7bTueqOVkqSrZ+xJ255H9yLDWm ztJ4bclCpEDyR5tk6DlCNbJtHNUpACZByOcT9w6wd7djK8VGid/QvupdljiQT1FSnaNxLp GZ/ufhu1TVLsjKg/JUHefCt/YsBh8s49654ad1rlaIkCI8nQtAcmLKiUqLDfcSzBJREAxV 4181T/X9pW2WrP3PtmPjfFJ6DwO12pZD7n0TeXk5RizitR9+dIRsUlPbbTh+VQ== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PDcY94MlDzN2W; Sat, 11 Feb 2023 17:10:05 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host81-158-36-31.range81-158.btcentralplus.com [81.158.36.31]) by smtp.theravensnest.org (Postfix) with ESMTPSA id A3E25105D0; Sat, 11 Feb 2023 17:10:03 +0000 (GMT) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: CFT: snmalloc as libc malloc From: David Chisnall In-Reply-To: <20230210162350.gg5g7dihp3zef3ov@mutt-hbsd> Date: Sat, 11 Feb 2023 17:10:02 +0000 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: <193D1E1C-73EB-42BB-8A6A-87B0E4AD9D7C@FreeBSD.org> References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> <20230210162350.gg5g7dihp3zef3ov@mutt-hbsd> To: Shawn Webb X-Mailer: Apple Mail (2.3654.120.0.1.13) X-ThisMailContainsUnwantedMimeParts: N On 10 Feb 2023, at 16:23, Shawn Webb wrote: >=20 > So I took a little bit of a different approach, which should provide > the same end result as your submodule approach. Note that I'm doing > this in HardenedBSD 14-CURRENT (the hardened/current/master branch). >=20 > 1. git cherry-pick -xs a5c83c69817d03943b8be982dd815c7e263d1a83 > 2. git rm -f .gitmodules contrib/snmalloc > 3. git commit > 4. git subtree add -P contrib/snmalloc \ > git@github.com:microsoft/snmalloc.git main >=20 > I believe this should leave me with a tree that populates > contrib/snmalloc and pulls in your non-contrib/ changes, leading me to > end up in the same end state as your submodule approach. >=20 > I am seeing some build errors. I've uploaded a WITHOUT_CLEAN=3Dyes log > to: >=20 > https://hardenedbsd.org/~shawn/2023-02-10_snmalloc-01.log.txt >=20 > Note that this is with llvm 15.0.7 that just landed in FreeBSD main. The error is a bit confusing, because nullptr_t has been in libc++ since = C++11. Does HardenedBSD change anything in include orders? Can you add = -v to the end of the compile command: c++ -target x86_64-unknown-freebsd14.0 = --sysroot=3D/usr/obj/data/src/hardenedbsd/amd64.amd64/tmp = -B/usr/obj/data/src/hardenedbsd/amd64.amd64/tmp/usr/bin = -fomit-frame-pointer -O2 -pipe -fno-common -DHARDENEDBSD -DNO__SCCSID = -DNO__RCSID -I/data/src/hardenedbsd/lib/libc/include = -I/data/src/hardenedbsd/include -I/data/src/hardenedbsd/lib/libc/amd64 = -DNLS -ftls-model=3Dinitial-exec -D__DBINTERFACE_PRIVATE = -I/data/src/hardenedbsd/contrib/gdtoa = -I/data/src/hardenedbsd/contrib/libc-vis -DINET6 = -I/usr/obj/data/src/hardenedbsd/amd64.amd64/lib/libc = -I/data/src/hardenedbsd/lib/libc/resolv -D_ACL_PRIVATE -DPOSIX_MISTAKE = -I/data/src/hardenedbsd/lib/libmd = -I/data/src/hardenedbsd/lib/libc/locale -DBROKEN_DES -DPORTMAP = -DDES_BUILTIN -I/data/src/hardenedbsd/lib/libc/rpc -DWANT_HYPERV -DYP = -DNS_CACHING -DSYMBOL_VERSIONING -g -gz=3Dzlib -mretpoline -fPIC -flto = -MD -MF.depend.malloc.o -MTmalloc.o -Wno-format-zero-length = -fstack-protector-strong -Wsystem-headers -Werror -Wall -Wno-format-y2k = -Wno-uninitialized -Wdate-time -Wno-empty-body -Wno-string-plus-int = -Wno-unused-const-variable -Wno-error=3Dunused-but-set-variable = -Wno-error=3Darray-parameter -Wno-error=3Ddeprecated-non-prototype = -Wno-error=3Dunused-but-set-parameter -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function = -Wno-enum-conversion -Wno-unused-local-typedef = -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum = -Wno-knr-promoted-parameter -Qunused-arguments = -I/data/src/hardenedbsd/lib/libutil = -I/data/src/hardenedbsd/lib/msun/amd64 = -I/data/src/hardenedbsd/lib/msun/x86 --include-directory-after = /data/src/hardenedbsd/lib/msun/src -DHARDENEDBSD = -I/data/src/hardenedbsd/contrib/snmalloc/src/snmalloc -std=3Dc++20 = -mcx16 -fno-exceptions -fno-rtti -DSNMALLOC_USE_THREAD_CLEANUP = -DSNMALLOC_BOOTSTRAP_ALLOCATOR -DSNMALLOC_JEMALLOC3_EXPERIMENTAL = -DSNMALLOC_JEMALLOC_NONSTANDARD -DSNMALLOC_PLATFORM_HAS_GETENTROPY = -DSNMALLOC_STATIC_LIBRARY_PREFIX=3D__ -ftls-model=3Dinitial-exec = -DSNMALLOC_CHECK_CLIENT -DSNMALLOC_FAIL_FAST=3Dfalse -DNDEBUG -g = -gz=3Dzlib -mretpoline -flto -Wno-c++11-extensions -c = /data/src/hardenedbsd/lib/libc/stdlib/snmalloc/malloc.cc -o malloc.o That should dump the include paths. It=E2=80=99s possible that the = libc++ headers are being included in the wrong order with respect to the = C paths? David= From nobody Sat Feb 11 17:13:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PDcdV4YCFz3pgBM for ; Sat, 11 Feb 2023 17:13:50 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PDcdV41T5z4Jmn; Sat, 11 Feb 2023 17:13:50 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676135630; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=phlCIFXv7OxSOsuPmRtD6Ny3VM/sCFvMIk1s6Edlizs=; b=MGSt9ZcHveN7hv6Mn/o5TRuMQckndVXQOIY6Ry1D3O/f3GaAlFOsdApQqBTQsCkMf2UHVD tcNG2NiVMkL+i16jkvNOO3OIySWJqC0ngCiev4xwo4zqfLfmy5AhCzF3+XBnO1yUxf49Nf VxFg4QwqXUlkX7is5///rN7GOzGtON1xb6iGXZsZvDU9OJz+sZYFZdwG0EpkKonf5vJJA5 b6dog1ZI3pd8S+u+IjfkDWMQk6FmGPEsWlFd/xsBj5fD+qm5ZF+yT7QILz35OWi6qZXZ2F cljb//8mVV6vmrJCJ+hHhC0JwSD/cuLlH7aB2ePga74iuTNbcdmIlHPbPaBfqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1676135630; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=phlCIFXv7OxSOsuPmRtD6Ny3VM/sCFvMIk1s6Edlizs=; b=Kx/paIIjFo/MABybAsc5ul9SgF3iJtATwT6DRtoGKDv0P+C35kRW1F7FOny4Oyki7Cb664 xMzoXmAT7KLKghR1+GZL43TVIaGdkGpm9+EQIrBHoeVZeqpsGKhMyInuEwMnttsXbuSjLk arHbPoH8CUByvbyj/B1J2nEWrhL9MbfGeeChBQZuWG+znwjkh6NpqKMWPx/+Yjf9aZazsE GvDA4NVSAOEaSqmfyXubP0JpizFP3O0FaH+yQ4RUACS2P27dqeLXjbFvy44Ez92G9MpK4j xXbRQW9yu/PLqThScgKVUA9jz9ZMtNm59I4J0WIOEoaRfH83aD6uQshcRakHYw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1676135630; a=rsa-sha256; cv=none; b=DX0Mh1fZqNKyr7y3TV8hqxnz9jYjHpK9rXEt2C7mei+OTgjH+FKFlFbNCFnu6zllvRh5vG I2uCxynWmHAFivwETTk5nbscc1IuyXfnCSPPkilIrpY2vJMVpy2N1tpJI/z+tjGVvMVb4j ZahvmAWGML+IAqPUppPI5U0vv+xzU4r5tlflkj40U3XyOOH+c3Qh6tyxURRWXDDS3VYxel 00fOP3xEzcZUMcNIZ9dmU8BMI05ryhMWs76d++NUvaKBeWO4dk06A3GCyVKCxSxCQQbxrU maIn9d042D+GPsiU2arv6R31b9hTr+xr9oW8jdk/MPJev+sGwnJ3AdvEu54z8Q== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PDcdV2wmmzNCc; Sat, 11 Feb 2023 17:13:50 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtpclient.apple (host81-158-36-31.range81-158.btcentralplus.com [81.158.36.31]) by smtp.theravensnest.org (Postfix) with ESMTPSA id C6A5F105D1; Sat, 11 Feb 2023 17:13:49 +0000 (GMT) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: CFT: snmalloc as libc malloc (the source_location issue) From: David Chisnall In-Reply-To: <96652C51-EAB7-46C7-9039-557A5DCDAB75@yahoo.com> Date: Sat, 11 Feb 2023 17:13:48 +0000 Cc: FreeBSD Hackers , shawn.webb@hardenedbsd.org, Dimitry Andric Content-Transfer-Encoding: quoted-printable Message-Id: References: <22F8AD1A-C1F9-48FA-BCED-E165E2DA5260@yahoo.com> <1982263E-57E0-4709-AAB3-DBB4F75C7546@yahoo.com> <96652C51-EAB7-46C7-9039-557A5DCDAB75@yahoo.com> To: Mark Millard X-Mailer: Apple Mail (2.3654.120.0.1.13) X-ThisMailContainsUnwantedMimeParts: N On 10 Feb 2023, at 21:34, Mark Millard wrote: >=20 > # find / -name source_location -print | more > = /usr/obj/DESTDIRs/main-amd64-chroot/usr/local/lib/gcc12/include/c++/experi= mental/source_location > = /usr/obj/DESTDIRs/main-amd64-chroot/usr/local/lib/gcc12/include/c++/source= _location > /usr/local/lib/gcc12/include/c++/experimental/source_location > /usr/local/lib/gcc12/include/c++/source_location >=20 > So, none for FreeBSD and its llvm15. >=20 > This makes sense, https://libcxx.llvm.org/Status/Cxx20.html shows: >=20 > P1208R6 LWG Adopt source_location for C++20 Cologne Complete 16.0 >=20 > So, likely FreeBSD will not have this until it progresses to > LLVM16 . It just changed to LLVM15 in main [so: FreeBSD 14]. The include of source_location is guarded under an #if __has_include, it = should be used only if it exists. If it doesn=E2=80=99t, there=E2=80=99s = a stub implementation. If you have GCC includes in your include path, = is it possible that it=E2=80=99s finding a source_location that is then = guarded behind a check for a compiler builtin that clang doesn=E2=80=99t = have? David