Date: Wed, 24 Feb 2021 17:27:42 -0700 From: Alan Somers <asomers@freebsd.org> To: Konstantin Belousov <kostikbel@gmail.com> Cc: Ravi Pokala <rpokala@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Message-ID: <CAOtMX2iDXoYYm=eMs12Rj-oQH3zVr15UW9htpAGajAWEWT-6%2Bg@mail.gmail.com> In-Reply-To: <YDbukaGGS%2BLh9zI9@kib.kiev.ua> References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> <YDbmBmFDN4J9/Zze@kib.kiev.ua> <CAOtMX2iHjjy6CiEkX91T2de9FvKj8wjhfuZdTNUemZ2a_Lsasw@mail.gmail.com> <YDbukaGGS%2BLh9zI9@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 24, 2021 at 5:26 PM Konstantin Belousov <kostikbel@gmail.com> wrote: > On Wed, Feb 24, 2021 at 04:55:46PM -0700, Alan Somers wrote: > > On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov <kostikbel@gmail.com > > > > wrote: > > > > > On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > > > > Hi folks, > > > > > > > > A colleague and I both independently observed `sysctl -a' appear to > hang > > > on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, > and ^C > > > didn't kill it. We could still establish a new terminal session to the > > > node, via SSH or serial console, and we were able to see that it was > > > actually spinning, not hung, and was consuming an entire CPU. > > > > > > > > We eventually determined that it was specifically `sysctl > > > vm.pmap.kernel_maps' which was spinning, and subsequently that it only > > > spinned if nvdimm.ko was loaded. It was not necessary to access the > device > > > node associated with the NVDIMM; merely having the module loaded was > > > sufficient. > > > > > > > > I know nvdimm(4) isn't terribly widely used, but hopefully someone > who > > > uses it can at least confirm my findings on this. Help in debugging > would > > > be even more appreciated. > > > > > > > > > > How large your nvdimms are? Their' SPAs are mapped into KVA fully and > this > > > could be quite large. It could be busy dumping page tables. > > > > > > Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over > it). > > > > > > > Speaking of vm.pmap.kernel_maps, that thing is huge. It easily dwarfs > all > > other sysctls combined, and tends to grow with time. Would it be > possible > > to hide it from sysctl -a's output? I think there are other sysctls like > > that, that are treated as opaque binary values. I once had to fix a bug > in > > py37-salt that caused a common operation to take _6_hours_ as opposed to > < > > 1 minute because of a huge vm.pmap.kernel_maps value, coupled with some > > O(n^2) string processing. > > jhb already marked this sysctl ask CTLFLAG_SKIP, several months ago. > The change was not merged back to 12. > Ahh, good.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2iDXoYYm=eMs12Rj-oQH3zVr15UW9htpAGajAWEWT-6%2Bg>