Date: Thu, 20 Nov 2025 08:01:01 -0600 From: Friedrich Doku <friedrichdoku2030@u.northwestern.edu> To: Mark Johnston <markj@freebsd.org> Cc: freebsd-hackers@freebsd.org Subject: Re: Kernel panic when using pmap_extract_and_hold() to check physical-to-virtual address mapping Message-ID: <CAD2_vGrnU47tsfaCVBJfSf1eMZH=xAxLdn4GVRhD3T71N7m-ww@mail.gmail.com> In-Reply-To: <CAD2_vGrEWrM=hRJQUG8xeYhfBtLvvgKLtUrJ3moBag6hPi=xgw@mail.gmail.com> References: <CAD2_vGoBqWmt%2BLgtTDOsJO9T_a2PdqZHjusv6EQeCRUDExxm1Q@mail.gmail.com> <aR4JUjIZxR1dPh-m@nuc> <CAD2_vGrEWrM=hRJQUG8xeYhfBtLvvgKLtUrJ3moBag6hPi=xgw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hello, I wanted to follow up on the pmap issue I reported earlier where userspace process pmaps appeared to have corrupted/invalid fields (pm_l0, pm_l0_paddr=0x0, garbage values in pm_stage and pm_levels). The issue was caused by building my kernel module outside the kernel source tree using a custom Makefile. This resulted in structure layout mismatches with my module seeing a different version or layout of struct pmap than what the kernel actually had, causing what appeared to be corrupted pointers but was actually just misaligned field access. After moving my module to sys/modules/ and building it, the pmap structure is now correctly aligned and all fields have valid values and pmap_extract() works correctly. I understand that there is no good method for building modules outside the kernel on FreeBSD. --Friedy On Wed, Nov 19, 2025 at 12:29 PM Friedrich Doku < friedrichdoku2030@u.northwestern.edu> wrote: > Thank you for your response. > > It happens every time we run the scan. Also, we're not checking for these > flags. Our test process is a simple user space program that mmaps memory > and waits, so P_WEXIT shouldn't be set, but we're not explicitly filtering > for it. > > It could be possible that I'm getting vmspace0. Maybe it's returning that, > but I haven't checked. Here's my full code <https://pastebin.com/7NnTFXtv>. > I'll run this again, so I can get the output from the crash. > --Friedrich > > > > > On Wed, Nov 19, 2025 at 12:15 PM Mark Johnston <markj@freebsd.org> wrote: > >> On Wed, Nov 19, 2025 at 11:51:02AM -0600, Friedrich Doku wrote: >> > Hello, >> > >> > We want to see if a specific physical address is mapped into the virtual >> > address space of a user space process. We are trying to do this from the >> > kernel, but we are running into issues with trying to use pmap_extract, >> > specifically we get the following kernel panic: >> > >> > panic: mtx_lock() of spin mutex (invalid) >> >> Does it always happen, or just sometimes? >> >> > The pmap pointer comes from: >> > >> > 1. pfind(target_pid) - gets the process structure >> >> Is it possible that one of P_WEXIT or P_SYSTEM is set in the p_flag >> field of the process? >> >> > 2. p->p_vmspace - gets the vmspace from the process >> >> Is it possible that p->p_vmspace == &vmspace0? >> >> > 3. vmspace_pmap(p->p_vmspace) - extracts the pmap from vmspace >> > >> > Then I'm iterating through vm_map entries with VM_MAP_ENTRY_FOREACH() >> and >> > calling pmap_extract_and_hold(pmap, va, VM_PROT_READ) for each virtual >> > address. >> >> Unrelated to your question, but this seems very slow. Each physical >> page carries a list of virtual mappings which refer to it, the "PV >> entry" list. You could instead look up the page by physical address >> (PHYS_TO_VM_PAGE()) and then walk the PV entry list for the page to find >> its mappings. This comes with a couple of caveats: >> - PV entries are implemented in machine dependent code, i.e., in the >> pmap layer, so some of your code would need to live there too. >> - PV entries don't record all mappings, just those that are mapped into >> user address spaces via so-called "managed" mappings. >> >> > The crash happens when calling pmap_extract_and_hold(). I suspect it's >> > trying to acquire pmap locks that conflict with something. >> > >> > I'm trying to find which virtual address maps to a given physical >> address >> > in a user space process. I'm doing this from a kernel module via sysctl >> > handler. >> > >> > Best, >> > Friedy >> > [-- Attachment #2 --] <div dir="ltr"><p class="gmail-font-claude-response-body gmail-whitespace-normal gmail-break-words">Hello,</p> <p class="gmail-font-claude-response-body gmail-whitespace-normal gmail-break-words">I wanted to follow up on the pmap issue I reported earlier where userspace process pmaps appeared to have corrupted/invalid fields (pm_l0, pm_l0_paddr=0x0, garbage values in pm_stage and pm_levels).</p> <p class="gmail-font-claude-response-body gmail-whitespace-normal gmail-break-words">The issue was caused by building my kernel module outside the kernel source tree using a custom Makefile. This resulted in structure layout mismatches with my module seeing a different version or layout of <code class="gmail-bg-text-200/5 gmail-border gmail-border-0.5 gmail-border-border-300 gmail-text-danger-000 gmail-whitespace-pre-wrap gmail-rounded-[0.4rem] gmail-px-1 gmail-py-px gmail-text-[0.9rem]">struct pmap</code> than what the kernel actually had, causing what appeared to be corrupted pointers but was actually just misaligned field access.</p><p class="gmail-font-claude-response-body gmail-whitespace-normal gmail-break-words">After moving my module to <code class="gmail-bg-text-200/5 gmail-border gmail-border-0.5 gmail-border-border-300 gmail-text-danger-000 gmail-whitespace-pre-wrap gmail-rounded-[0.4rem] gmail-px-1 gmail-py-px gmail-text-[0.9rem]">sys/modules/</code> and building it, the pmap structure is now correctly aligned and all fields have valid values and <code class="gmail-bg-text-200/5 gmail-border gmail-border-0.5 gmail-border-border-300 gmail-text-danger-000 gmail-whitespace-pre-wrap gmail-rounded-[0.4rem] gmail-px-1 gmail-py-px gmail-text-[0.9rem]">pmap_extract()</code> works correctly. I understand that there is no good method for building modules outside the kernel on FreeBSD.</p><p class="gmail-font-claude-response-body gmail-whitespace-pre-wrap gmail-break-words">--Friedy</p><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Wed, Nov 19, 2025 at 12:29 PM Friedrich Doku <<a href="mailto:friedrichdoku2030@u.northwestern.edu">friedrichdoku2030@u.northwestern.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you for your response. <p>It happens every time we run the scan. Also, we're not checking for these flags. Our test process is a simple user space program that mmaps memory and waits, so P_WEXIT shouldn't be set, but we're not explicitly filtering for it. </p><p>It could be possible that I'm getting vmspace0. Maybe it's returning that, but I haven't checked. Here's my full <a href="https://pastebin.com/7NnTFXtv" target="_blank">code</a>. I'll run this again, so I can get the output from the crash. </p>--Friedrich<p><br></p><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 19, 2025 at 12:15 PM Mark Johnston <<a href="mailto:markj@freebsd.org" target="_blank">markj@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, Nov 19, 2025 at 11:51:02AM -0600, Friedrich Doku wrote:<br> > Hello,<br> > <br> > We want to see if a specific physical address is mapped into the virtual<br> > address space of a user space process. We are trying to do this from the<br> > kernel, but we are running into issues with trying to use pmap_extract,<br> > specifically we get the following kernel panic:<br> > <br> > panic: mtx_lock() of spin mutex (invalid)<br> <br> Does it always happen, or just sometimes?<br> <br> > The pmap pointer comes from:<br> > <br> > 1. pfind(target_pid) - gets the process structure<br> <br> Is it possible that one of P_WEXIT or P_SYSTEM is set in the p_flag<br> field of the process?<br> <br> > 2. p->p_vmspace - gets the vmspace from the process<br> <br> Is it possible that p->p_vmspace == &vmspace0?<br> <br> > 3. vmspace_pmap(p->p_vmspace) - extracts the pmap from vmspace<br> > <br> > Then I'm iterating through vm_map entries with VM_MAP_ENTRY_FOREACH() and<br> > calling pmap_extract_and_hold(pmap, va, VM_PROT_READ) for each virtual<br> > address.<br> <br> Unrelated to your question, but this seems very slow. Each physical<br> page carries a list of virtual mappings which refer to it, the "PV<br> entry" list. You could instead look up the page by physical address<br> (PHYS_TO_VM_PAGE()) and then walk the PV entry list for the page to find<br> its mappings. This comes with a couple of caveats:<br> - PV entries are implemented in machine dependent code, i.e., in the<br> pmap layer, so some of your code would need to live there too.<br> - PV entries don't record all mappings, just those that are mapped into<br> user address spaces via so-called "managed" mappings.<br> <br> > The crash happens when calling pmap_extract_and_hold(). I suspect it's<br> > trying to acquire pmap locks that conflict with something.<br> > <br> > I'm trying to find which virtual address maps to a given physical address<br> > in a user space process. I'm doing this from a kernel module via sysctl<br> > handler.<br> > <br> > Best,<br> > Friedy<br> </blockquote></div> </blockquote></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD2_vGrnU47tsfaCVBJfSf1eMZH=xAxLdn4GVRhD3T71N7m-ww>