From nobody Fri Nov 21 19:08:47 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dClB91RlJz6HcfY for ; Fri, 21 Nov 2025 19:08:49 +0000 (UTC) (envelope-from mmel@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R13" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dClB870Q5z3Xvb; Fri, 21 Nov 2025 19:08:48 +0000 (UTC) (envelope-from mmel@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763752129; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Wj7idLNjevG2NGXbFSeizKCDpmztYXzj5ktua28yK98=; b=hVUq20HzGyXfF+BfIUL2xIYjbyWFyQdzsg+XCQN5V72YqYIYlXH8ZMm5MEWWs+a7XYlu1w /SDg/lHrFM4FCwx7FLeC4uCQXG2ONrnzjK5I/YhwrhrmwegaiZzBxmb8TDs0Y1j6XkQNTu BZjzAF2mzTwp8RgHmcXv28anbzkpYIXIXBJnvYTBzKS0YTO3YAE343+xpx4cwgQ1aMmhX6 I3xgJ8aPs7qcNpuTK5F3Aqj3tNj4Gpv0IMkaN7afy9ohZg3Ych/3PysiHG2FnT2i7SCZi3 DBISh51ReuXNQdZs0YK5HOLx3Zobk+8qYa+edUedEp9vmYrsp6AEFkC8jqsByA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1763752129; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Wj7idLNjevG2NGXbFSeizKCDpmztYXzj5ktua28yK98=; b=QW2ajW8Ys/RQUyqB9bAG7EzzlTa/1UqJznWr7HK4K9al/HrrnDuBKYPO6nWpStrDsMY4XV 4DMIhgJSW+uF+/M06i14z73mfn88wC5V04j+td+k8NP70JqYgYA454Pop7plJ1i6NylrwK A61Agv4LoGMb89fsu3zOIdijhlC0wX4tbAH5Iq1aCgS9mQ5da2XgsRDRiABHNC8W9fT8Fb 9UYkWvjzpvjlx0EGf/U8K9B/Hx5tKINSCKQWNtjMAobM1OYFHJxTvLc0AdQidtTjtrLwjy 5psnKA5QVAjOdTYE/egb1JynNF7rCsXCRUXV88VSWtTXriyoKYkkGZTHijwrkQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1763752129; a=rsa-sha256; cv=none; b=vjuALm7LYNlnw90JAhVaiI3zogFt39XiiWRz3frxG+k5GTh1mJswZnvZwn8U8zhHcPtZ+6 FOkTN6h1P8QJLJbBGkyY9g2rdj1lZK9TFLAUNofvrdgYb+XhkTdc/dOaIfHVd3e0U1vJdI vNIQ1CQR8z7h4kyyduh/HzF+QL667KO7zkgzXhqY6yUu4BxOHNkc0OifVpX4FrvMRTQcLw TUG94EMAOC13GPd3Y66J3z2UstyOx8wJEbKfoYE/4IAkUL0y4BhW879/983Pk0esTC/bJB 3rN2jXienE5lg2+gOKYNp76ZYSs8luD62+9fSODPijv+NCsJKxuZ5PLUkWfT4g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [192.168.168.195] (internet-251.radiolinkplus.cz [109.205.241.251]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mmel/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4dClB83t52z7Hf; Fri, 21 Nov 2025 19:08:48 +0000 (UTC) (envelope-from mmel@freebsd.org) Message-ID: Date: Fri, 21 Nov 2025 20:08:47 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Michal Meloun Reply-To: mmel@FreeBSD.org Subject: Re: mmap( MAP_ANON) is broken on current. (was Still seeing Failed assertion: "p[i] == 0" on armv7 buildworld) To: Konstantin Belousov Cc: FreeBSD Current References: <8657a2f4-cb32-49a5-bbf6-cd5a4394c7be@FreeBSD.org> Content-Language: cs, en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 21.11.2025 10:03, Konstantin Belousov wrote: > On Fri, Nov 21, 2025 at 10:36:42AM +0200, Konstantin Belousov wrote: >> On Fri, Nov 21, 2025 at 08:12:55AM +0100, Michal Meloun wrote: >>> I have confirmed that jmalloc assertions are caused by mmap() failure. It >>> can return non-zeroed page(s) for mmap(MAP_ANON), which is clearly a bug. >>> >>> I have confirmed this on native ARMv7, and according to Mark, it is also >>> reproducible on ARM32 and i386 jails. I think I saw it also on a >>> memory-constrained (4 GB) aarch64, but I cannot reproduce it yet. >>> >>> Have somebody idea how to identify vm faults associated with anon mmap to >>> trigger detection of this failure in kernel? Or any other hint? >> >> I think It would be much more visible if freshly allocated anonymous pages >> are corrupted. A similar mechanism to get zeroed pages is used to get >> fresh page table pages, and corruption there must cause a lot of kernel >> page faults with 'invalid PTE bit' hw reports. >> But of course everything is possible. >> >> VM has an optimization where we track known-to-be-zeroed free page >> separately, by marking them with PG_ZERO flag. If allocation needs a >> zeroed page and the flag is set, we skip calling pmap_zero_page() on it. >> >> Also, in vm_page_free_prep() when we are told that the page is zeroed, >> with DIAGNOSTIC enabled, on amd64 and arm64, we do check for that. >> >> So lets add slow check for vm_fault code that supposedly zeroed page is >> indeed zeroed. Can you try to catch the issue with the patch applied, >> and DIAGNOSTIC enabled? Patch is arch-agnostic and I believe should >> work on armv7, although obviously causing slowdown. > > I also made the vm_page_free_prep() check MI. > Please use https://reviews.freebsd.org/D53850 instead of the previous > patch. Hi Kib, i was unexpectedly out of the office today, so I only got back to debugging a moment ago and couldn't devote much time to it today. First, many thanks for your efforts, but this check doesn't trigger when the problem occurs To be more precise, testing case on fresh kernel(d8bfcacd12aba73188c44a157c707908e275825d) with PMAP_DEBUG defined in pmap-v6.c and with trivial zero check for first page at this place -> https://cgit.freebsd.org/src/tree/contrib/jemalloc/src/pages.c#n281 causes this failure: __je_pages_map: addr: 0x0, ret: 0x3087b000, size: 4096, alignment: 4096, prot: 0x00000003, flags: 0x0C001002 __je_pages_map: i: 0, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 23, p[i]: 0x308E5F94, p: 0x3087b000 __je_pages_map: i: 27, p[i]: 0x308E23F4, p: 0x3087b000 __je_pages_map: i: 29, p[i]: 0x308F077C, p: 0x3087b000 __je_pages_map: i: 30, p[i]: 0x308C3444, p: 0x3087b000 __je_pages_map: i: 33, p[i]: 0x308C57BC, p: 0x3087b000 __je_pages_map: i: 36, p[i]: 0x308E41E4, p: 0x3087b000 __je_pages_map: i: 39, p[i]: 0x308EA2E4, p: 0x3087b000 __je_pages_map: i: 42, p[i]: 0x308EC444, p: 0x3087b000 __je_pages_map: i: 44, p[i]: 0x308EE60C, p: 0x3087b000 __je_pages_map: i: 47, p[i]: 0x308C7AF4, p: 0x3087b000 __je_pages_map: i: 58, p[i]: 0x308C9F24, p: 0x3087b000 __je_pages_map: i: 79, p[i]: 0x308E8114, p: 0x3087b000 __je_pages_map: i: 80, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 160, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 240, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 320, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 400, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 480, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 560, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 640, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 720, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 800, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 880, p[i]: 0xFFFFFFFF, p: 0x3087b000 __je_pages_map: i: 960, p[i]: 0xFFFFFFFF, p: 0x3087b000 The pattern looks interesting; it is not exactly same in all cases, but it is similar. Another example: __je_pages_map: addr: 0x0, ret: 0x32d4d000, size: 4096, alignment: 4096, prot: 0x00000003, flags: 0x0C001002 __je_pages_map: i: 64, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 144, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 224, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 304, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 384, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 408, p[i]: 0x32CE5854, p: 0x32d4d000 __je_pages_map: i: 416, p[i]: 0x32CE5064, p: 0x32d4d000 __je_pages_map: i: 429, p[i]: 0x32CE5BD4, p: 0x32d4d000 __je_pages_map: i: 455, p[i]: 0x32CE5FD4, p: 0x32d4d000 __je_pages_map: i: 464, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 544, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 589, p[i]: 0x32CE6744, p: 0x32d4d000 __je_pages_map: i: 591, p[i]: 0x32CE6B04, p: 0x32d4d000 __je_pages_map: i: 603, p[i]: 0x32CE6EC4, p: 0x32d4d000 __je_pages_map: i: 624, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 704, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 784, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 864, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 944, p[i]: 0xFFFFFFFF, p: 0x32d4d000 __je_pages_map: i: 969, p[i]: 0x40F8917C, p: 0x32d4d000 __je_pages_map: i: 971, p[i]: 0x40EB9F0C, p: 0x32d4d000 __je_pages_map: i: 973, p[i]: 0x40D164CC, p: 0x32d4d000 __je_pages_map: i: 978, p[i]: 0x40F47EFC, p: 0x32d4d000 __je_pages_map: i: 980, p[i]: 0x4116768C, p: 0x32d4d000 __je_pages_map: i: 996, p[i]: 0x3E5430FC, p: 0x32d4d000 __je_pages_map: i: 1002, p[i]: 0x40F88FAC, p: 0x32d4d000 __je_pages_map: i: 1006, p[i]: 0x40D1669C, p: 0x32d4d000 __je_pages_map: i: 1011, p[i]: 0x40F47D2C, p: 0x32d4d000 __je_pages_map: i: 1012, p[i]: 0x3E542F2C, p: 0x32d4d000 __je_pages_map: i: 1021, p[i]: 0x40EB9D3C, p: 0x32d4d000 __je_pages_map: i: 1022, p[i]: 0x4116785C, p: 0x32d4d000 Still searching, Michal