From nobody Sun Oct 16 15:23:52 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mr3nB3mWYz4fRk0 for ; Sun, 16 Oct 2022 15:23:58 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mr3n95d6Nz3XcS for ; Sun, 16 Oct 2022 15:23:57 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x729.google.com with SMTP id d13so5355583qko.5 for ; Sun, 16 Oct 2022 08:23:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=ThvMBI6sisUhEnISi/GUPTsglHmecS8elyQqROFjxU8=; b=oc+9Ie7si2gEEOqG7FJ7RiUK8Ql1y+okrGXwzMtoPlblrpO2xON8yWWzH6caxtWhaN 7c+bkOPz2ALYhOE6hHZg2i2kYpIgI1C3vOsi9WroUXv+Pt99JOYr0M9h3YZDqT+OvaDI 0H/Hqdv3bwNNNtd0O9GK08Z1D2C9bJ+fbYfujerBnysu+llz9TbNUXrugRxfwRmp55bv lji7dX18/JUymMLw3EsomtzGLE41nhGpSxa3us5bZPiDw1gIMXQ1udJby2U0SROgrptb aV5LAe9rzP0yxZSXwKTj9VkmTGH9rQFb3W9yjAdDGkvjJDc98r+EYCY6fIZn+JDweT8J U2VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ThvMBI6sisUhEnISi/GUPTsglHmecS8elyQqROFjxU8=; b=ymKrK2UmZsFza+Vwx1Qddnzwz22voFPqcKzrJwNBAGiDP0SwV9d65HKVoQgZ5aXskf SyCprGrbQin/bCHOEl2FvMn/weXb4nCg7EubyFtH2zV9i8fKXqiXJiF2UxzQ4YqxsIVM lHkd0PoyGuIuj4R0VTJXj2rdysDkIw3p5/dTEM84Ajz45FyYjx6foR8TNvT/og9HTTRl MEwLnrWqqPVEBe1Vf6bziN3wk/0YnEyvJRBFnDAcRUDjjpgLGSms+aDf219oMwFsGCEt AB0tiMgDOiX0+u/vI/VkXaU+POGLK1XwtIpuWsjX/8/hJWuZ454aO9zV7NoIP9njS6PX 0BOA== X-Gm-Message-State: ACrzQf3+9EMpgXMpsN4VyG5sDR4b7D5tSJLy9euJVpVxPOlNJeXjMV7p dSZ6p7OZMPAuwCzkYD6bZW/LQlfj3AI= X-Google-Smtp-Source: AMsMyM4dZfwlpbWvLunelSjEvU1aPhsi0ZzUnTdRTJsJzWjDMDN5wLb1RCvzWNJX4yv0oYFF5MLHiw== X-Received: by 2002:a05:620a:2995:b0:6ee:e3a0:9fb8 with SMTP id r21-20020a05620a299500b006eee3a09fb8mr1123265qkp.165.1665933836463; Sun, 16 Oct 2022 08:23:56 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id k6-20020a378806000000b006ea5a9984d1sm7101566qkd.94.2022.10.16.08.23.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 08:23:54 -0700 (PDT) Date: Sun, 16 Oct 2022 11:23:52 -0400 From: Mark Johnston To: Paul Floyd Cc: freebsd-hackers Subject: Re: AMD64 14.0-CURRENT memory layout changes Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4Mr3n95d6Nz3XcS X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=oc+9Ie7s; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::729 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.70 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::729:from]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; FREEMAIL_TO(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Sun, Oct 16, 2022 at 05:08:30PM +0200, Paul Floyd wrote: > Hi > > I just noticed that the memory layout has changed for elf binaries > running on amd64 (my last attempt to setup an i386 VM failed so I can't > confirm if that also changed, and I'm not yet concerned by other platforms). > > Here's a procstat -v for ksh93 on 13.1 on the host machine > > > paulf> procstat -v 1456 > > PID START END PRT RES PRES REF SHD FLAG TP PATH > > 1456 0x200000 0x273000 r-- 70 343 30 16 CN--- vn /usr/local/bin/ksh93 > > 1456 0x273000 0x3d1000 r-x 257 343 30 16 CN--- vn /usr/local/bin/ksh93 > > 1456 0x3d1000 0x3dd000 r-- 11 0 1 0 CN--- vn /usr/local/bin/ksh93 > > 1456 0x3dd000 0x3de000 rw- 1 0 1 0 CN--- vn /usr/local/bin/ksh93 > > 1456 0x3de000 0x3e4000 rw- 6 0 1 0 C---- vn /usr/local/bin/ksh93 > > 1456 0x3e4000 0x3ec000 rw- 5 5 1 0 C---- df > > 1456 0x8003de000 0x8003e4000 r-- 6 28 364 120 CN--- vn /libexec/ld-elf.so.1 > > 1456 0x8003e4000 0x8003fb000 r-x 23 28 364 120 CN--- vn /libexec/ld-elf.so.1 > > 1456 0x8003fb000 0x8003fc000 r-- 1 0 1 0 CN--- vn /libexec/ld-elf.so.1 > > 1456 0x8003fc000 0x800411000 rw- 19 19 1 0 CN--- df > > 1456 0x800420000 0x800432000 r-- 13 33 352 176 CN--- vn /lib/libm.so.5 > > 1456 0x800432000 0x800459000 r-x 20 33 352 176 CN--- vn /lib/libm.so.5 > > 1456 0x800459000 0x80045a000 rw- 1 0 1 0 CN--- vn /lib/libm.so.5 > > 1456 0x80045a000 0x80045b000 rw- 1 0 1 0 CN--- vn /lib/libm.so.5 > > 1456 0x80045b000 0x8004df000 r-- 132 380 528 284 CN--- vn /lib/libc.so.7 > > 1456 0x8004df000 0x80062b000 r-x 234 380 528 284 CN--- vn /lib/libc.so.7 > > 1456 0x80062b000 0x800633000 r-- 8 0 1 0 CN--- vn /lib/libc.so.7 > > 1456 0x800633000 0x800634000 rw- 1 0 1 0 CN--- vn /lib/libc.so.7 > > 1456 0x800634000 0x80063b000 rw- 7 0 1 0 C---- vn /lib/libc.so.7 > > 1456 0x80063b000 0x8008f5000 rw- 84 84 1 0 C---- df > > 1456 0x800a00000 0x801200000 rw- 13 13 1 0 CN--- df > > 1456 0x7fffdffff000 0x7ffffffdf000 --- 0 0 0 0 ----- gd > > 1456 0x7ffffffdf000 0x7ffffffff000 rw- 11 11 1 0 C--D- df > > 1456 0x7ffffffff000 0x800000000000 r-x 1 1 125 0 ----- ph > > Here the stack starts at 0x7ffffffdf000 > > And the same on 14.0 running on a 4Gbyte VirtualBox VM > > > paulf@freebsd:~/valgrind $ procstat -v 62770 > > PID START END PRT RES PRES REF SHD FLAG TP PATH > > 62770 0x200000 0x273000 r-- 115 488 4 2 CN--- vn /usr/local/bin/ksh93 > > 62770 0x273000 0x3c7000 r-x 340 488 4 2 CN--- vn /usr/local/bin/ksh93 > > 62770 0x3c7000 0x3d4000 r-- 13 0 2 0 C---- vn /usr/local/bin/ksh93 > > 62770 0x3d4000 0x3d5000 rw- 1 0 2 0 C---- vn /usr/local/bin/ksh93 > > 62770 0x3d5000 0x3da000 rw- 5 0 1 0 C---- vn /usr/local/bin/ksh93 > > 62770 0x3da000 0x3e2000 rw- 5 5 1 0 ----- sw > > 62770 0x80075d000 0x82073d000 --- 0 0 0 0 ----- gd > > 62770 0x82073d000 0x82075d000 rw- 14 14 1 0 ---D- sw > > 62770 0x8209c8000 0x8209c9000 r-x 1 1 28 0 ----- ph > > 62770 0x8217b0000 0x8217c2000 rw- 16 16 1 0 ----- sw > > 62770 0x822186000 0x822210000 r-- 138 496 104 54 CN--- vn /lib/libc.so.7 > > 62770 0x822210000 0x82235e000 r-x 334 496 104 54 CN--- vn /lib/libc.so.7 > > 62770 0x82235e000 0x822367000 r-- 9 0 2 0 C---- vn /lib/libc.so.7 > > 62770 0x822367000 0x822368000 rw- 1 0 2 0 C---- vn /lib/libc.so.7 > > 62770 0x822368000 0x82236f000 rw- 7 0 1 0 C---- vn /lib/libc.so.7 > > 62770 0x82236f000 0x82259e000 rw- 20 20 1 0 ----- sw > > 62770 0x823434000 0x823447000 r-- 19 59 4 2 CN--- vn /lib/libm.so.5 > > 62770 0x823447000 0x82346f000 r-x 40 59 4 2 CN--- vn /lib/libm.so.5 > > 62770 0x82346f000 0x823470000 rw- 1 0 1 0 C---- vn /lib/libm.so.5 > > 62770 0x823470000 0x823471000 rw- 1 0 1 0 C---- vn /lib/libm.so.5 > > 62770 0x823e0e000 0x823e3e000 rw- 16 16 1 0 ----- sw > > 62770 0x824600000 0x824800000 rw- 11 11 1 0 ----- sw > > 62770 0x8251a1000 0x8253a1000 rw- 1 1 1 0 ----- sw > > 62770 0x825e00000 0x826200000 rw- 3 3 1 0 ----- sw > > 62770 0x826a49000 0x826a61000 rw- 8 8 1 0 ----- sw > > 62770 0x826e5c000 0x826e74000 rw- 14 14 1 0 ----- sw > > 62770 0x827d6e000 0x827d86000 rw- 9 9 1 0 ----- sw > > 62770 0x8288ba000 0x8288d2000 rw- 5 5 1 0 ----- sw > > 62770 0x8296db000 0x8296f3000 rw- 3 3 1 0 ----- sw > > 62770 0xeeeecc15000 0xeeeecc1b000 r-- 6 29 71 21 CN--- vn /libexec/ld-elf.so.1 > > 62770 0xeeeecc1b000 0xeeeecc32000 r-x 23 29 71 21 CN--- vn /libexec/ld-elf.so.1 > > 62770 0xeeeecc32000 0xeeeecc33000 r-- 1 0 1 0 C---- vn /libexec/ld-elf.so.1 > > 62770 0xeeeecc33000 0xeeeecc35000 rw- 2 2 1 0 ----- sw > > 62770 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd > > > ldrt is now mapped up at 0xeeeecc15000 and the user stack looks like it > starts at 0x82073d000. > > This is causing me problems with Valgrind, which creates the guest stack > at 0x7ffffffdf000. > > I haven't yet done any debugging of the problem but this causes > > Fatal error 'Cannot allocate red zone for initial thread' at line 395 in > file /usr/src/lib/libthr/thread/thr_init.c (errno = 22) > > for elf binaries linked with libthr.so > > Can anyone point me to more information on this change? Phabricator for > instance. It is related to some changes to randomize the base of the process stack, as we do for other components of the address space. See https://cgit.freebsd.org/src/commit/?id=1811c1e957ee1250b08b3246fc0db37ddf64b736 for example. In 14.0 randomization is enabled by default. > Are there any syscalls that control where rtld gets loaded and/or where > the stack base is located? No, their locations are randomized. There are sysctl knobs for controlling whether randomization is enabled or not. > Also is there a sysctl to disable this changed mapping, as a temporary > workaround? Setting kern.elf(64|32).aslr.stack to 0 should restore the old behaviour. It should also be possible to disable this on a per-process basis with proccontrol(1), but that doesn't appear to work, i.e., there is a bug. However, all randomization can be disabled this way, try "procstat -m aslr -s disable ksh93".