From owner-freebsd-current@freebsd.org Wed Apr 7 20:56:51 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DB2FE5C4839 for ; Wed, 7 Apr 2021 20:56:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4FFxXM4vkGz4lyL for ; Wed, 7 Apr 2021 20:56:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id A665C5C473D; Wed, 7 Apr 2021 20:56:51 +0000 (UTC) Delivered-To: current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A616F5C47D8; Wed, 7 Apr 2021 20:56:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FFxXM486fz4m1t; Wed, 7 Apr 2021 20:56:51 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x736.google.com with SMTP id c3so36479qkc.5; Wed, 07 Apr 2021 13:56:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=VhZcu/rCADQ3skTtkI5k9QMHJgXlzwNTnFYTmqgB/+s=; b=DUacyMJ3iB1WSKX8PQpah2MZKR51rRDK8bgVXbT+Pc0R++QZDle20gJ8cxuP/zUpad tfNDQ8B2zwqZJ2VqMb/rXy5Co4B1eRqIlNnROUQdKJQb5p7I/tDPWUqZV75WYnZQhkmz Ku2hjQohQ5oUcVcMQi6HW47flnU95yLRzcejEx8utVvn4MIQZjaeONsTJiv700d+GiMe Xa75u4CRbgwgybMRE/zlKTPEV1Qoyhixo7OnsseS+7SXLSxOpml0PYJ/5LigoBlkww5B 4vsmEupjcAMi30ysXpgT1D25mHEjzivZho+yBwDNjkm9b2isRhWJEUm8SZMhxYZlZ1hP b65g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=VhZcu/rCADQ3skTtkI5k9QMHJgXlzwNTnFYTmqgB/+s=; b=X+bTaUq6uleYcYX+RaOtJ3i+NM4AB5HyLCu0AfgUlmgh29kFe23bK1pYoKCj5sy0ET A3bp0cTeSsEGzPHMqb85QDCwn8RBYz/DhBHmgsrpXwBrMRfvd9tOmsO17B8HgfRHBjwV 0J5F6DXMqCHcpg+2T0Vpe/z9aT99Ht0QRZNN3Anuc8tiv/T4l7is0GPelO/CVNK1Sf6k eHkV+A9lYek1OGxCCNHcwFfEFFSqZUlz87RviWZrcbwa2zmILtQrTjhMQnehC9+cuMRi IwAIFPQmWVY9yVzrVh+2Ri2n0+Iybt5eediv5sUXt0HZiG1M3kNrEXqnUfB8vofl2B7O liUA== X-Gm-Message-State: AOAM530BQIL6SFhKI1amJWybqDeDtv/uGgFgO2tr/727PKBb6lvaogKo tvisA8cNqKJBeJbUwoG/WVu+6SxdZ+wfpQ== X-Google-Smtp-Source: ABdhPJw8KJCJCrmNfirAO0R55fh49f2WzW5yo4AXcfbPY7/QoOsxPmEIMQdRRwauUM10C4DT0s+CSQ== X-Received: by 2002:a37:a8cc:: with SMTP id r195mr5083749qke.86.1617829010163; Wed, 07 Apr 2021 13:56:50 -0700 (PDT) Received: from nuc ([142.126.164.150]) by smtp.gmail.com with ESMTPSA id m8sm9582660qtg.67.2021.04.07.13.56.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Apr 2021 13:56:49 -0700 (PDT) Sender: Mark Johnston Date: Wed, 7 Apr 2021 16:56:49 -0400 From: Mark Johnston To: Andriy Gapon Cc: freebsd-stable List , FreeBSD Current Subject: Re: stable/13, vm page counts do not add up Message-ID: References: <0606571f-fec0-c7ad-98e8-a0b8554918e2@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4FFxXM486fz4m1t X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Apr 2021 20:56:51 -0000 On Wed, Apr 07, 2021 at 11:22:41PM +0300, Andriy Gapon wrote: > On 07/04/2021 22:54, Mark Johnston wrote: > > On Wed, Apr 07, 2021 at 10:42:57PM +0300, Andriy Gapon wrote: > >> > >> I regularly see that the top's memory line does not add up (and by a lot). > >> That can be seen with vm.stats as well. > >> > >> For example: > >> $ sysctl vm.stats | fgrep count > >> vm.stats.vm.v_cache_count: 0 > >> vm.stats.vm.v_user_wire_count: 3231 > >> vm.stats.vm.v_laundry_count: 262058 > >> vm.stats.vm.v_inactive_count: 3054178 > >> vm.stats.vm.v_active_count: 621131 > >> vm.stats.vm.v_wire_count: 1871176 > >> vm.stats.vm.v_free_count: 187777 > >> vm.stats.vm.v_page_count: 8134982 > >> > >> $ bc > >>>>> 187777 + 1871176 + 621131 + 3054178 + 262058 > >> 5996320 > >>>>> 8134982 - 5996320 > >> 2138662 > >> > >> As you can see, it's not a small number of pages either. > >> Approximately 2 million pages, 8 gigabytes or 25% of the whole memory on this > >> system. > >> > >> This is 47c00a9835926e96, 13.0-STABLE amd64. > >> I do not think that I saw anything like that when I used (much) older FreeBSD. > > > > One relevant change is that vm_page_wire() no longer removes pages from > > LRU queues, so the count of pages in the queues can include wired pages. > > If the page daemon runs, it will dequeue any wired pages that are > > encountered. > > Maybe I misunderstand how that works, but I would expect that the sum of all > counters could be greater than v_page_count at times. But in my case it's less. I misread, sorry. You're right, what I described would cause double counting. I don't know what might be causing it then. It could be a page leak. The kernel allocates wired pages without adjusting the v_wire_count counter in some cases, but the ones I know about happen at boot and should not account for such a large disparity. I do not see it on a few systems that I have access to. > > This was done to reduce queue lock contention, operations like > > sendfile() which transiently wire pages would otherwise trigger two > > queue operations per page. Now that queue operations are batched this > > might not be as important. > > > > We could perhaps add a new flavour of vm_page_wire() which is not lazy > > and would be suited for e.g., the buffer cache. What is the primary > > source of wired pages in this case? > > It should be ZFS, I guess. > > -- > Andriy Gapon