From owner-freebsd-hackers@freebsd.org Wed Feb 24 23:55:59 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 44F0754D888 for ; Wed, 24 Feb 2021 23:55:59 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DmCVR1TDyz4pG3; Wed, 24 Feb 2021 23:55:59 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f42.google.com with SMTP id c16so4034223otp.0; Wed, 24 Feb 2021 15:55:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cj3VZ3pVoqIO+Fvrh7C9l44ieHnc+14znhk/bQWQUYQ=; b=ufIUTIALsC3iYS0U03nADUSDPeso3pFe7PTUIS08760GpunRBg7ee/0jxgeKqmpuxN CR+GBDU7ZnrK5adE4gS+rx0FhOW8+2iNcKtbIMp19uIVZEY4fsC86dJuaJYAgdKxdu0k 6+roAoGaQZANjdPLQ8gJmjOVmGqs9pyqWDTxLoDERXIc6M70azxOS1LLtsmMU08xEK+O a6UE7qRrwRX8xQkMcf23O46DHqoAQ45R7x8SsEMla8cc9p4jJ9boFshidMRZUiTR9KfF jZcMC6WxNEmokieWaGcxYEt7cBvAHFTyBbOwcb+YgYf9njf4Db+391TAaOQ7Gt8816pw 81MA== X-Gm-Message-State: AOAM532kzgZsQ4drCMMiXGqP1wFtmlWZyPnzZywChiOsUxQZ/lY3HS91 5r5TFnVOILMJYXqZrZG6KGepZOwtpaMTEoFUri4= X-Google-Smtp-Source: ABdhPJx6e/MZ6x654tlXL9cubXA/rXZfw7zrO/oRAmErXpoZubwyc+YoxxeSRUzpHZdg3WNLm0+QOeEwO+Q7+1/emsM= X-Received: by 2002:a9d:3642:: with SMTP id w60mr64287otb.18.1614210958000; Wed, 24 Feb 2021 15:55:58 -0800 (PST) MIME-Version: 1.0 References: <2C7B4C6A-0432-44A6-B512-D7114F2B092B@freebsd.org> In-Reply-To: From: Alan Somers Date: Wed, 24 Feb 2021 16:55:46 -0700 Message-ID: Subject: Re: `sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko To: Konstantin Belousov Cc: Ravi Pokala , "freebsd-hackers@freebsd.org" X-Rspamd-Queue-Id: 4DmCVR1TDyz4pG3 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 23:55:59 -0000 On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov wrote: > On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote: > > Hi folks, > > > > A colleague and I both independently observed `sysctl -a' appear to hang > on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C > didn't kill it. We could still establish a new terminal session to the > node, via SSH or serial console, and we were able to see that it was > actually spinning, not hung, and was consuming an entire CPU. > > > > We eventually determined that it was specifically `sysctl > vm.pmap.kernel_maps' which was spinning, and subsequently that it only > spinned if nvdimm.ko was loaded. It was not necessary to access the device > node associated with the NVDIMM; merely having the module loaded was > sufficient. > > > > I know nvdimm(4) isn't terribly widely used, but hopefully someone who > uses it can at least confirm my findings on this. Help in debugging would > be even more appreciated. > > > > How large your nvdimms are? Their' SPAs are mapped into KVA fully and this > could be quite large. It could be busy dumping page tables. > > Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over it). > Speaking of vm.pmap.kernel_maps, that thing is huge. It easily dwarfs all other sysctls combined, and tends to grow with time. Would it be possible to hide it from sysctl -a's output? I think there are other sysctls like that, that are treated as opaque binary values. I once had to fix a bug in py37-salt that caused a common operation to take _6_hours_ as opposed to < 1 minute because of a huge vm.pmap.kernel_maps value, coupled with some O(n^2) string processing.