From owner-freebsd-hackers@freebsd.org Thu Feb 25 00:26:01 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0CAC954E730 for ; Thu, 25 Feb 2021 00:26:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmD9458R7z4rY5; Thu, 25 Feb 2021 00:26:00 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 11P0PrxP021161 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 25 Feb 2021 02:25:56 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 11P0PrxP021161 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 11P0Prwd021160; Thu, 25 Feb 2021 02:25:53 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 25 Feb 2021 02:25:53 +0200 From: Konstantin Belousov To: Alan Somers Cc: Ravi Pokala , "freebsd-hackers@freebsd.org" Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko Message-ID: References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DmD9458R7z4rY5 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2021 00:26:01 -0000 On Wed, Feb 24, 2021 at 04:55:46PM -0700, Alan Somers wrote: > On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov > wrote: > > > On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > > > Hi folks, > > > > > > A colleague and I both independently observed `sysctl -a' appear to hang > > on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C > > didn't kill it. We could still establish a new terminal session to the > > node, via SSH or serial console, and we were able to see that it was > > actually spinning, not hung, and was consuming an entire CPU. > > > > > > We eventually determined that it was specifically `sysctl > > vm.pmap.kernel_maps' which was spinning, and subsequently that it only > > spinned if nvdimm.ko was loaded. It was not necessary to access the device > > node associated with the NVDIMM; merely having the module loaded was > > sufficient. > > > > > > I know nvdimm(4) isn't terribly widely used, but hopefully someone who > > uses it can at least confirm my findings on this. Help in debugging would > > be even more appreciated. > > > > > > > How large your nvdimms are? Their' SPAs are mapped into KVA fully and this > > could be quite large. It could be busy dumping page tables. > > > > Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over it). > > > > Speaking of vm.pmap.kernel_maps, that thing is huge. It easily dwarfs all > other sysctls combined, and tends to grow with time. Would it be possible > to hide it from sysctl -a's output? I think there are other sysctls like > that, that are treated as opaque binary values. I once had to fix a bug in > py37-salt that caused a common operation to take _6_hours_ as opposed to < > 1 minute because of a huge vm.pmap.kernel_maps value, coupled with some > O(n^2) string processing. jhb already marked this sysctl ask CTLFLAG_SKIP, several months ago. The change was not merged back to 12.