From nobody Tue Aug 3 14:30:38 2021 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 52AFB12DE531; Tue, 3 Aug 2021 14:30:44 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GfHNN1fNNz4cSl; Tue, 3 Aug 2021 14:30:44 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qt1-x82f.google.com with SMTP id l24so14059665qtj.4; Tue, 03 Aug 2021 07:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=1M5fQDgkCEFNwsznOMTBmNzq/CgEBTJM5Dxyj0+urYY=; b=bWo+jelDhTilbtTqARRM9YkAWPliiDlZhqe9R0NcVaXxFtjQ/XVIXTzU60Onr6mS3v YYvLUpGZq13udovaFV5vO2kGm4JeEoE1M6GiHfLJoWrf34yso9Exzd7AJ7MZ2uGFB1cd zyMCjZzt5ElwSI05OFggxK8FBGfnrn3Ikd4rTCZpm/GkiOzo2tyzRid11o9mwBp1W3Zi qA48f4WExhVqyIg/4sw7hsAPj5xwBvVfJzY7tFOGSeWp0XpslfbXcDV76AFJULfMS/T1 njI4Y3T2Z5LzwLpw851d3AwB9u+tYyg1cHNfM2CAlANgEnfJ6A/XbLbcJYGCjBmK8r5l O7tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=1M5fQDgkCEFNwsznOMTBmNzq/CgEBTJM5Dxyj0+urYY=; b=SUNJgg6VMFXrV2LDC4iupPZzDXnWT0u16cGBiVfDvr3n18xKjFNMg3iKkRON+smHMr SdaPalie7RIJ2IYZf6cdIlCMmQ0o+HUjdtLv+9Ex4QVyzW/+qVkpaK3FFtFBaNuurqiV 3NWc3MKCJXswRy7AGJZZs6NGzkJDlRZb8YPXfviW4tzaTDpMEzCJhiJ5UrqeNArGnRMb cw+oco81WZ/Ppqz8HOWoaLgmX6CjCYVdARzCZPPs3w1yEzLOUSl/XDF63jd+Mhz59V8m Okx7S02JG5/PMZoKTWMAyOwCMFSXqKmSVAMn45HttuCj5Ffhi/zMk6Gy9ID3aGTZx+ZQ RxwQ== X-Gm-Message-State: AOAM533kC4SQks3q2TX631mh68b2ZunSEwYqhNtMvkWyeaPg0lXWJs9y bPjpm7bvsSbfwK2T0iRBswQ= X-Google-Smtp-Source: ABdhPJxNfl6yVgq8dxHie8bFKvV6sj8nZm65OoLg7vG6yIQetMGerppUupzyYf0YlD/pIyqpalqhdw== X-Received: by 2002:a05:622a:14ce:: with SMTP id u14mr18659018qtx.208.1628001037852; Tue, 03 Aug 2021 07:30:37 -0700 (PDT) Received: from nuc ([142.126.162.193]) by smtp.gmail.com with ESMTPSA id j2sm6105912qtn.46.2021.08.03.07.30.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Aug 2021 07:30:37 -0700 (PDT) Date: Tue, 3 Aug 2021 10:30:38 -0400 From: Mark Johnston To: "Andrey V. Elsukov" Cc: =?iso-8859-1?Q?=D6zkan?= KIRIK , FreeBSD Net , freebsd-stable Subject: Re: Wired Memory Increasing about 500MBytes per day Message-ID: References: <5dc957ec-9483-0a80-b29e-be4b71c1b9d9@yandex.ru> <35e9249e-37f7-3ff8-23c5-a22d8b909f09@yandex.ru> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <35e9249e-37f7-3ff8-23c5-a22d8b909f09@yandex.ru> X-Rspamd-Queue-Id: 4GfHNN1fNNz4cSl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-Spam: Yes X-ThisMailContainsUnwantedMimeParts: N On Tue, Aug 03, 2021 at 04:59:34PM +0300, Andrey V. Elsukov wrote: > 03.08.2021 16:47, Mark Johnston пишет: > >> We noticed the same problem, I'm not sure the exact version, but you can > >> check the output: > >> # vmstat -z | egrep "ITEM|pgcache" > >> > >> The page cache grows until lowmem is not reached. Then it automatically > >> cleans and begins to grow again. > > > > The pgcache zones simply provide a per-CPU cache and allocator for > > physical page frames. The sizes of the caches are bounded. The numbers > > of "used" items from the pgcache zones do not really tell you anything > > since those pages may be allocated for any number of purposes, including > > for other UMA zones. For instance, if ZFS allocates a buffer page from > > its ABD UMA zone, and that zone's caches are empty, UMA may allocate a > > new slab using uma_small_alloc() -> vm_page_alloc() -> pgcache zone. > > > > So if there is some wired page leak, the pgcache zones are probably not > > directly responsible. > > We don't see any leaks, but our monitoring shows that "free" memory > migrates to "wired" and only these zones are grow. How are you measuring this? USED or USED+FREE? > So, we have on the > graphs linear growing of wired memory over 7 days. When free memory > reaches ~4% all returns to normal, and then again linear growing for 7 > days. And pgcache zones reset their number of USED items to low value. > This is on the server with 256G RAM. > > E.g. This is when 9% of free memory left: > > $ vmstat -z | egrep "ITEM|pgcache" > ITEM SIZE LIMIT USED FREE REQ > FAILSLEEP XDOMAIN > vm pgcache: 4096, 0, 5225, 139, 412976, 0, > 0, 0 > vm pgcache: 4096, 0,28381269, 77,190108006, 24, > 0, 0 > vm pgcache: 4096, 0, 166358, 11523,1684567513,3054, > 0, 0 > vm pgcache: 4096, 0,29548679, 576,780034183,1730, > 0, 0 > $ bc > >>> 5225+28381269+166358+29548679 > 58101531 > >>> 58101531*4096/1024/1024/1024 > 221 > >>> > > This is when lowmem triggered: > % vmstat -z | egrep "ITEM|pgcache" > ITEM SIZE LIMIT USED FREE REQ > FAILSLEEP XDOMAIN > vm pgcache: 4096, 0, 5336, 337, 410052, 0, > 0, 0 > vm pgcache: 4096, 0, 3126129, 117,56689945, 24, > 0, 0 > vm pgcache: 4096, 0, 49771, 3910,413657845,1828, > 0, 0 > vm pgcache: 4096, 0, 4249924, 706,224519238, 562, > 0, 0 > % bc > >>> 5336+3126129+49771+4249924 > 7431160 > >>> 7431160*4096/1024/1024/1024 > 28 > >>> > > Look at the graph: > https://imgur.com/yhqK1p8.png > > -- > WBR, Andrey V. Elsukov >