Date: Wed, 24 Feb 2021 16:55:46 -0700 From: Alan Somers <asomers@freebsd.org> To: Konstantin Belousov <kostikbel@gmail.com> Cc: Ravi Pokala <rpokala@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Message-ID: <CAOtMX2iHjjy6CiEkX91T2de9FvKj8wjhfuZdTNUemZ2a_Lsasw@mail.gmail.com> In-Reply-To: <YDbmBmFDN4J9/Zze@kib.kiev.ua> References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> <YDbmBmFDN4J9/Zze@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov <kostikbel@gmail.com> wrote: > On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > > Hi folks, > > > > A colleague and I both independently observed `sysctl -a' appear to hang > on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C > didn't kill it. We could still establish a new terminal session to the > node, via SSH or serial console, and we were able to see that it was > actually spinning, not hung, and was consuming an entire CPU. > > > > We eventually determined that it was specifically `sysctl > vm.pmap.kernel_maps' which was spinning, and subsequently that it only > spinned if nvdimm.ko was loaded. It was not necessary to access the device > node associated with the NVDIMM; merely having the module loaded was > sufficient. > > > > I know nvdimm(4) isn't terribly widely used, but hopefully someone who > uses it can at least confirm my findings on this. Help in debugging would > be even more appreciated. > > > > How large your nvdimms are? Their' SPAs are mapped into KVA fully and this > could be quite large. It could be busy dumping page tables. > > Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over it). > Speaking of vm.pmap.kernel_maps, that thing is huge. It easily dwarfs all other sysctls combined, and tends to grow with time. Would it be possible to hide it from sysctl -a's output? I think there are other sysctls like that, that are treated as opaque binary values. I once had to fix a bug in py37-salt that caused a common operation to take _6_hours_ as opposed to < 1 minute because of a huge vm.pmap.kernel_maps value, coupled with some O(n^2) string processing.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2iHjjy6CiEkX91T2de9FvKj8wjhfuZdTNUemZ2a_Lsasw>