From nobody Thu Nov 20 14:01:01 2025 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dC0Pl5yp6z6HRBM for ; Thu, 20 Nov 2025 14:01:15 +0000 (UTC) (envelope-from friedrichdoku2030@u.northwestern.edu) Received: from mail-oa1-x2c.google.com (mail-oa1-x2c.google.com [IPv6:2001:4860:4864:20::2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dC0Pk4nysz49TK for ; Thu, 20 Nov 2025 14:01:14 +0000 (UTC) (envelope-from friedrichdoku2030@u.northwestern.edu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=u-northwestern-edu.20230601.gappssmtp.com header.s=20230601 header.b=j1h6buE+; dmarc=pass (policy=none) header.from=northwestern.edu; spf=pass (mx1.freebsd.org: domain of friedrichdoku2030@u.northwestern.edu designates 2001:4860:4864:20::2c as permitted sender) smtp.mailfrom=friedrichdoku2030@u.northwestern.edu Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-3e0ad5b4763so368505fac.1 for ; Thu, 20 Nov 2025 06:01:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u-northwestern-edu.20230601.gappssmtp.com; s=20230601; t=1763647272; x=1764252072; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=426eRMRcns/6aXn6RWx4KL1OwWrHfq+Gw/0AoU9fR0A=; b=j1h6buE+PqYa4v6I2z7JKsSZrGUMgv9nbZHB4/EB/6cLgMRUbxmf6V9A7Ac9yuonCf nL2SPVKz5eN4LitA4TP/Eop5hCPKAeVlelZDbvpVOowWwEO9ovUbf1x/OXCt7zCeo/ZI XpaggsEDrLvg7uqzmuR+BYGpp8tuCxqvfUZ+Lw54gX8HcfU3wlXptCjuEEPP5cZFzhhh gftcKqAmXSdRsmSNFRI3YNzlNIA4guBXzSF3nCe6pRCGGjyA9bwA9bKU57eaCnkwSoeQ AMGb36X+GHcUIC70B6+nKUxWlCtQ5lA/R5CxRCW9eShEj9lQrDkY7KmtTTUFsRDQSkLD 2o1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763647272; x=1764252072; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=426eRMRcns/6aXn6RWx4KL1OwWrHfq+Gw/0AoU9fR0A=; b=G4kt8uJP3TgSYYZQkMeN+kFmwdcn5afsylK3DHItepLq9i/NQoPvedutVHfiokyN2n 74txHymDeuBDvwPFQznjaOxu4ErQZ7h8jHVhHTvPPWLARepwEDSbrDt3v1/j8RnzVzV2 7IIkPFHwT29jb3ZZyM6uWlgNMCzrEarurEkrN6FXu5DQEJiHoNbw7KtfEvNIu/n4HbEp jR08rg/FbCEW6GKQVIizyubNJBEFiUZ+qajvK7M677xjUB1RWw/0Mgiv0fjJaGiiwElZ Sh3H8vRpgrt5zK7UXjooDjoaphGaJ5pPsXVQCX53RAsQ62qoDPPyRNLMjIkTQ8147eOh OiDQ== X-Gm-Message-State: AOJu0YxZTkVz2IKkWLmXibwqXXKFTNSKapH9Cx8cqBZr40zgJXq0VYPs p2GDBbH9WU1f1jsqX0Nik8vWEA7u//v9wN0xov9gDYxjoQAyVK2ekoq612xk10H8DZMc7kW8h83 tbpUHim6If5qcN4M8toXf7/JQNIT86ctoPfuH8XkOgKsPGj8SJ8a6/rGH9A== X-Gm-Gg: ASbGncvzgj+ThbyHyeysVfKR2E42MzTVjJduq1qxCSMdDenuLdcBZbEK1P6BrWa6Uq4 jHFUJNANfwxxqJOHtsMOeHufpsWrtIOZa6bvgx6bfb2XY+sEcUo59ij8h3V9uDmBQVIhmN9VOEm HRRqtDuKOyD1n+lytYRNxZsfcKUD0rs8iFcV/CHuOkBjElKRFH7TKbSv2aGF8ma+wgxuWI7vEht 2DXMdzTpaeYvy1P8sfzez9gOp9c356eLD5Yihj50CDE70YQzIgY5P6kWjbJQe4WQld+Nd7ofegp dmChw6zKRXp7p4spWrmWcUElIEos+vYS6S+fCfQ9RWhUC4sJpWQ1H+4ctJGkVNHxgc0XiutLy7D dfulMT1K2CGL6oA== X-Google-Smtp-Source: AGHT+IEd0qaPaRpgfeHAamNLIc2iG39kbaoH3hz15KjaJVirHXrMdC3rkMlLJSYCwNSxlZ65kes6YicbKljUE6WqefU= X-Received: by 2002:a05:6820:6403:b0:657:47c6:5a6f with SMTP id 006d021491bc7-65782ba633dmr602787eaf.3.1763647271686; Thu, 20 Nov 2025 06:01:11 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Friedrich Doku Date: Thu, 20 Nov 2025 08:01:01 -0600 X-Gm-Features: AWmQ_bmLt46f36jrmjvEa3HITpfZD4DMHQhIPY15N9YxmxpEht91FPDQZ7-O5Vw Message-ID: Subject: Re: Kernel panic when using pmap_extract_and_hold() to check physical-to-virtual address mapping To: Mark Johnston Cc: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="00000000000023646b064407216f" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; DMARC_POLICY_ALLOW(-0.50)[northwestern.edu,none]; R_DKIM_ALLOW(-0.20)[u-northwestern-edu.20230601.gappssmtp.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::2c:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[u-northwestern-edu.20230601.gappssmtp.com:+] X-Rspamd-Queue-Id: 4dC0Pk4nysz49TK --00000000000023646b064407216f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, I wanted to follow up on the pmap issue I reported earlier where userspace process pmaps appeared to have corrupted/invalid fields (pm_l0, pm_l0_paddr=3D0x0, garbage values in pm_stage and pm_levels). The issue was caused by building my kernel module outside the kernel source tree using a custom Makefile. This resulted in structure layout mismatches with my module seeing a different version or layout of struct pmap than what the kernel actually had, causing what appeared to be corrupted pointers but was actually just misaligned field access. After moving my module to sys/modules/ and building it, the pmap structure is now correctly aligned and all fields have valid values and pmap_extract(= ) works correctly. I understand that there is no good method for building modules outside the kernel on FreeBSD. --Friedy On Wed, Nov 19, 2025 at 12:29=E2=80=AFPM Friedrich Doku < friedrichdoku2030@u.northwestern.edu> wrote: > Thank you for your response. > > It happens every time we run the scan. Also, we're not checking for these > flags. Our test process is a simple user space program that mmaps memory > and waits, so P_WEXIT shouldn't be set, but we're not explicitly filterin= g > for it. > > It could be possible that I'm getting vmspace0. Maybe it's returning that= , > but I haven't checked. Here's my full code . > I'll run this again, so I can get the output from the crash. > --Friedrich > > > > > On Wed, Nov 19, 2025 at 12:15=E2=80=AFPM Mark Johnston wrote: > >> On Wed, Nov 19, 2025 at 11:51:02AM -0600, Friedrich Doku wrote: >> > Hello, >> > >> > We want to see if a specific physical address is mapped into the virtu= al >> > address space of a user space process. We are trying to do this from t= he >> > kernel, but we are running into issues with trying to use pmap_extract= , >> > specifically we get the following kernel panic: >> > >> > panic: mtx_lock() of spin mutex (invalid) >> >> Does it always happen, or just sometimes? >> >> > The pmap pointer comes from: >> > >> > 1. pfind(target_pid) - gets the process structure >> >> Is it possible that one of P_WEXIT or P_SYSTEM is set in the p_flag >> field of the process? >> >> > 2. p->p_vmspace - gets the vmspace from the process >> >> Is it possible that p->p_vmspace =3D=3D &vmspace0? >> >> > 3. vmspace_pmap(p->p_vmspace) - extracts the pmap from vmspace >> > >> > Then I'm iterating through vm_map entries with VM_MAP_ENTRY_FOREACH() >> and >> > calling pmap_extract_and_hold(pmap, va, VM_PROT_READ) for each virtual >> > address. >> >> Unrelated to your question, but this seems very slow. Each physical >> page carries a list of virtual mappings which refer to it, the "PV >> entry" list. You could instead look up the page by physical address >> (PHYS_TO_VM_PAGE()) and then walk the PV entry list for the page to find >> its mappings. This comes with a couple of caveats: >> - PV entries are implemented in machine dependent code, i.e., in the >> pmap layer, so some of your code would need to live there too. >> - PV entries don't record all mappings, just those that are mapped into >> user address spaces via so-called "managed" mappings. >> >> > The crash happens when calling pmap_extract_and_hold(). I suspect it's >> > trying to acquire pmap locks that conflict with something. >> > >> > I'm trying to find which virtual address maps to a given physical >> address >> > in a user space process. I'm doing this from a kernel module via sysct= l >> > handler. >> > >> > Best, >> > Friedy >> > --00000000000023646b064407216f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hello,

I wanted to follow up on the pmap issue I reported earlier wher= e userspace process pmaps appeared to have corrupted/invalid fields (pm_l0,= pm_l0_paddr=3D0x0, garbage values in pm_stage and pm_levels).

The issue was caused by building my kernel module outside the k= ernel source tree using a custom Makefile. This resulted in structure layou= t mismatches with my module seeing a different version or layout of struct pmap than what= the kernel actually had, causing what appeared to be corrupted pointers bu= t was actually just misaligned field access.

After moving my= module to sys/module= s/ and building it, the pmap structure is now correctly aligned and = all fields have valid values=C2=A0and pmap_extract() works correctly. I understand that t= here is no good method for building modules outside the kernel on FreeBSD.<= /p>

--Friedy



On Wed, Nov 19, 2025= at 12:29=E2=80=AFPM Friedrich Doku <friedrichdoku2030@u.northwestern.edu> wrote:
T= hank you for your response.=C2=A0

It happens every time we run the scan. = Also, we're not checking for these flags. Our test process is a simple = user space program that mmaps memory and waits, so P_WEXIT shouldn't be= set, but we're not explicitly filtering for it.=C2=A0

It could b= e possible that I'm getting vmspace0. Maybe it's returning that, bu= t I haven't checked. Here's my full code. I'll run this again, so I can ge= t the output from the crash.=C2=A0

--Friedrich




<= div class=3D"gmail_quote">
On Wed, Nov= 19, 2025 at 12:15=E2=80=AFPM Mark Johnston <markj@freebsd.org> wrote:
On Wed, Nov 19, 2025 at 11:51:= 02AM -0600, Friedrich Doku wrote:
> Hello,
>
> We want to see if a specific physical address is mapped into the virtu= al
> address space of a user space process. We are trying to do this from t= he
> kernel, but we are running into issues with trying to use pmap_extract= ,
> specifically we get the following kernel panic:
>
> panic: mtx_lock() of spin mutex (invalid)

Does it always happen, or just sometimes?

> The pmap pointer comes from:
>
>=C2=A0 =C2=A0 1. pfind(target_pid) - gets the process structure

Is it possible that one of P_WEXIT or P_SYSTEM is set in the p_flag
field of the process?

>=C2=A0 =C2=A0 2. p->p_vmspace - gets the vmspace from the process
Is it possible that p->p_vmspace =3D=3D &vmspace0?

>=C2=A0 =C2=A0 3. vmspace_pmap(p->p_vmspace) - extracts the pmap from= vmspace
>
> Then I'm iterating through vm_map entries with VM_MAP_ENTRY_FOREAC= H() and
> calling pmap_extract_and_hold(pmap, va, VM_PROT_READ) for each virtual=
> address.

Unrelated to your question, but this seems very slow.=C2=A0 Each physical page carries a list of virtual mappings which refer to it, the "PV
entry" list.=C2=A0 You could instead look up the page by physical addr= ess
(PHYS_TO_VM_PAGE()) and then walk the PV entry list for the page to find its mappings.=C2=A0 This comes with a couple of caveats:
- PV entries are implemented in machine dependent code, i.e., in the
=C2=A0 pmap layer, so some of your code would need to live there too.
- PV entries don't record all mappings, just those that are mapped into=
=C2=A0 user address spaces via so-called "managed" mappings.

> The crash happens when calling pmap_extract_and_hold(). I suspect it&#= 39;s
> trying to acquire pmap locks that conflict with something.
>
> I'm trying to find which virtual address maps to a given physical = address
> in a user space process. I'm doing this from a kernel module via s= ysctl
> handler.
>
> Best,
> Friedy
--00000000000023646b064407216f--