Date: Fri, 10 Feb 2023 11:23:50 -0500 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: David Chisnall <theraven@FreeBSD.org> Cc: freebsd-hackers <freebsd-hackers@freebsd.org> Subject: Re: CFT: snmalloc as libc malloc Message-ID: <20230210162350.gg5g7dihp3zef3ov@mutt-hbsd> In-Reply-To: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--7ww2svwck3f2m2k5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 09, 2023 at 12:08:49PM +0000, David Chisnall wrote: > Hi, >=20 > For the few yearsI've been running locally with snmalloc as the malloc in > libc. Eventually I'd like to propose this for upstreaming but it needs s= ome > wider testing first. >=20 > For those unfamiliar with snmalloc (https://github.com/microsoft/snmalloc= ), > it is an allocator (or, rather, a toolkit for building allocators) from my > team at Microsoft Research designed for both performance and security. A > few highlights: >=20 > - Snmalloc uses a message-passing design, which makes allocating on one > thread and freeing on another cheap. > - Very fast allocation performance > - Randomisation of relative locations of allocations > - Most metadata is stored out-of-band > - In-band metadata uses some lighweight encryption to protect against > corruption. > - Support for CHERI. >=20 > In the (limited!) testing that I've done, it outperforms jemalloc and > results in a smaller libc binary. >=20 > I've also previously managed to use it in the kernel, though that code > hasn't been tested in a while (last used with FreeBSD 11): >=20 > https://github.com/microsoft/snmalloc/blob/main/src/snmalloc/pal/pal_free= bsd_kernel.h >=20 > It is also used in the Verona process sandboxing work, which makes it easy > to isolate a library in a capsicum Sandbox: >=20 > https://github.com/microsoft/verona/tree/master/experiments/process_sandb= ox >=20 > We test on FreeBSD in CI upstream and the code is actively maintained. > We have implemented compatibility wrappers for all of the jemalloc > non-standard APIs that FreeBSD's libc exposes. >=20 > In particular, snmalloc is designed to make it very cheap to find the sta= rt > and end of an allocation, given a heap pointer. This means that we can > insert bounds checks in critical libc functions to prevent heap overflow. > This is done in the branch for memcpy, which some investigation of a corp= us > of security vulnerabilities showed was the root cause of about 10% of > arbitrary-code-execution vulnerabilities. >=20 > The bounds checks are controlled via an environment variable > LIBC_BOUNDS_CHECKS. Setting this to 0 disables checks, to 1 checks on > destination arguments, and to 2 checks sources and destinations. An ifunc > resolver selects the correct memcpy implementation at load time. >=20 > I did have a version that checked a bunch of other libc functions (e.g. > sprintf, puts) but it was quite hacky (and the way the ifunc resolves was > implemented broke tcl). >=20 > The current branch puts two things behind the MALLOC_PRODUCTION toggle: >=20 > - The additional security checks that detect corruption of malloc state. > - Pretty-printing errors. >=20 > We are currently separating the former into separate knobs upstream, some > subset should probably be turned on by default in production. The latter > has less of a performance impact than it had and will probably be on for = all > configurations at some point once we've refactored slightly to ensure the > compiler can tail call the failure function (which moves it entirely off = the > fast path). With this enabled, you get errors that look like this: >=20 > Fatal Error! > memcpy with source out of bounds of heap allocation: > range [0x14823c02440, 0x14823c0246a) > allocation [0x14823c02440, 0x14823c02450) > range goes beyond allocation by 0x1a bytes >=20 > Abort trap (core dumped) >=20 > Without it, you just get an illegal instruction trap. >=20 > There are a few limitations in the current branch: >=20 > - The memcpy integration is broken on non-amd64 platforms (patches welco= me > from people who can test these!). > - Only memcpy (not, for example, memmove) has bounds checks. > - The memcpy in rtld is naive, which may impact performance. > - MALLOC_PRODUCTION conflates too many things >=20 > The branch is here: >=20 > https://github.com/davidchisnall/freebsd-src/tree/snmalloc2 >=20 > It adds snmalloc as a submodule in contrib. FreeBSD is allergic to > submodules, so upstreaming will need to replace this with something more > complicated. You should be able to cherry-pick the top commit on any > vaguely-recent -CURRENT. So I took a little bit of a different approach, which should provide the same end result as your submodule approach. Note that I'm doing this in HardenedBSD 14-CURRENT (the hardened/current/master branch). 1. git cherry-pick -xs a5c83c69817d03943b8be982dd815c7e263d1a83 2. git rm -f .gitmodules contrib/snmalloc 3. git commit 4. git subtree add -P contrib/snmalloc \ git@github.com:microsoft/snmalloc.git main I believe this should leave me with a tree that populates contrib/snmalloc and pulls in your non-contrib/ changes, leading me to end up in the same end state as your submodule approach. I am seeing some build errors. I've uploaded a WITHOUT_CLEAN=3Dyes log to: https://hardenedbsd.org/~shawn/2023-02-10_snmalloc-01.log.txt Note that this is with llvm 15.0.7 that just landed in FreeBSD main. Any non-XKCD[138]-conforming pointers would be appreciated. ;-) Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --7ww2svwck3f2m2k5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmPmb48ACgkQ/y5nonf4 4foIoQ//Y90bPyFfbRmK8vPOs0cZ4EKWuui5YZvyvv1BxM7yU9VFWeVwF/C1HIP1 0MKkxRyxE3s68K/xMBj2VCx6OgBShofVlzIkz9RWiza39CliU5Qx0ZcSZbDRT5ng UXTCut/VTIbaNJtpZMYswl6tdMs4U1yroi8r9UsytmGkV9PREl51YDFTPnopVmSC m79/INU4IOdz0uee9LDZfx95G1fU/sgjjNL9dQtTHuDSrq5iOynILlfr1bcEoR1O IZAOp+P36AeqodHMDh+E7G/vO04qhR7MugHkNpUP/WQEIa0dLtffo58BWnvzCymd h9I8wtTl5RGyy0gkgnrQWdOEEz8GV6hIMDtr5/LrL/YHQUPC2xoF5cyvIh/4fmLr oxKY/GfCDu49wy5sHZ14iBM5v8zGESZK1nx19XX/vctpwV3xQkH6FnOqGFzX9Rkw vLR5AtGOSf8Wsk+4Doki3UjG0Ax+KzCWhQGkKEW2hNWHEz9k1oq0gBykURSedqEA UPqabFF/bc7TP3d6lec8OwFmaCjCKZBmLZxvG4K3tDYb+wqtYZ6EeNDJ29gamFmW lwxdHfOuPE/NrWx1f/Ie9a6xkTTM4o10397pOBqeFmEWzvAkz7k928ib0INxh1qP CAcfH7NaDFCAI2Vfeq2NckdRXdYwOqFFDmlDOi4sYQUdam/8LMU= =8Zb7 -----END PGP SIGNATURE----- --7ww2svwck3f2m2k5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20230210162350.gg5g7dihp3zef3ov>