Date: Thu, 25 Feb 2021 02:25:53 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: Alan Somers <asomers@freebsd.org> Cc: Ravi Pokala <rpokala@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Message-ID: <YDbukaGGS%2BLh9zI9@kib.kiev.ua> In-Reply-To: <CAOtMX2iHjjy6CiEkX91T2de9FvKj8wjhfuZdTNUemZ2a_Lsasw@mail.gmail.com> References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> <YDbmBmFDN4J9/Zze@kib.kiev.ua> <CAOtMX2iHjjy6CiEkX91T2de9FvKj8wjhfuZdTNUemZ2a_Lsasw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 24, 2021 at 04:55:46PM -0700, Alan Somers wrote: > On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov <kostikbel@gmail.com> > wrote: > > > On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > > > Hi folks, > > > > > > A colleague and I both independently observed `sysctl -a' appear to hang > > on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C > > didn't kill it. We could still establish a new terminal session to the > > node, via SSH or serial console, and we were able to see that it was > > actually spinning, not hung, and was consuming an entire CPU. > > > > > > We eventually determined that it was specifically `sysctl > > vm.pmap.kernel_maps' which was spinning, and subsequently that it only > > spinned if nvdimm.ko was loaded. It was not necessary to access the device > > node associated with the NVDIMM; merely having the module loaded was > > sufficient. > > > > > > I know nvdimm(4) isn't terribly widely used, but hopefully someone who > > uses it can at least confirm my findings on this. Help in debugging would > > be even more appreciated. > > > > > > > How large your nvdimms are? Their' SPAs are mapped into KVA fully and this > > could be quite large. It could be busy dumping page tables. > > > > Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over it). > > > > Speaking of vm.pmap.kernel_maps, that thing is huge. It easily dwarfs all > other sysctls combined, and tends to grow with time. Would it be possible > to hide it from sysctl -a's output? I think there are other sysctls like > that, that are treated as opaque binary values. I once had to fix a bug in > py37-salt that caused a common operation to take _6_hours_ as opposed to < > 1 minute because of a huge vm.pmap.kernel_maps value, coupled with some > O(n^2) string processing. jhb already marked this sysctl ask CTLFLAG_SKIP, several months ago. The change was not merged back to 12.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YDbukaGGS%2BLh9zI9>