Date: Tue, 3 Aug 2021 10:30:38 -0400 From: Mark Johnston <markj@freebsd.org> To: "Andrey V. Elsukov" <bu7cher@yandex.ru> Cc: =?iso-8859-1?Q?=D6zkan?= KIRIK <ozkan.kirik@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org>, freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: Wired Memory Increasing about 500MBytes per day Message-ID: <YQlTDhvqckfqAKUI@nuc> In-Reply-To: <35e9249e-37f7-3ff8-23c5-a22d8b909f09@yandex.ru> References: <CAAcX-AFWQrRjcniS5gKHzFuw3KKVJ-GKkFGmaA6OLwQ5vu9Jyg@mail.gmail.com> <5dc957ec-9483-0a80-b29e-be4b71c1b9d9@yandex.ru> <YQlI4cfsiCgH74WK@nuc> <35e9249e-37f7-3ff8-23c5-a22d8b909f09@yandex.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Aug 03, 2021 at 04:59:34PM +0300, Andrey V. Elsukov wrote: > 03.08.2021 16:47, Mark Johnston пишет: > >> We noticed the same problem, I'm not sure the exact version, but you can > >> check the output: > >> # vmstat -z | egrep "ITEM|pgcache" > >> > >> The page cache grows until lowmem is not reached. Then it automatically > >> cleans and begins to grow again. > > > > The pgcache zones simply provide a per-CPU cache and allocator for > > physical page frames. The sizes of the caches are bounded. The numbers > > of "used" items from the pgcache zones do not really tell you anything > > since those pages may be allocated for any number of purposes, including > > for other UMA zones. For instance, if ZFS allocates a buffer page from > > its ABD UMA zone, and that zone's caches are empty, UMA may allocate a > > new slab using uma_small_alloc() -> vm_page_alloc() -> pgcache zone. > > > > So if there is some wired page leak, the pgcache zones are probably not > > directly responsible. > > We don't see any leaks, but our monitoring shows that "free" memory > migrates to "wired" and only these zones are grow. How are you measuring this? USED or USED+FREE? > So, we have on the > graphs linear growing of wired memory over 7 days. When free memory > reaches ~4% all returns to normal, and then again linear growing for 7 > days. And pgcache zones reset their number of USED items to low value. > This is on the server with 256G RAM. > > E.g. This is when 9% of free memory left: > > $ vmstat -z | egrep "ITEM|pgcache" > ITEM SIZE LIMIT USED FREE REQ > FAILSLEEP XDOMAIN > vm pgcache: 4096, 0, 5225, 139, 412976, 0, > 0, 0 > vm pgcache: 4096, 0,28381269, 77,190108006, 24, > 0, 0 > vm pgcache: 4096, 0, 166358, 11523,1684567513,3054, > 0, 0 > vm pgcache: 4096, 0,29548679, 576,780034183,1730, > 0, 0 > $ bc > >>> 5225+28381269+166358+29548679 > 58101531 > >>> 58101531*4096/1024/1024/1024 > 221 > >>> > > This is when lowmem triggered: > % vmstat -z | egrep "ITEM|pgcache" > ITEM SIZE LIMIT USED FREE REQ > FAILSLEEP XDOMAIN > vm pgcache: 4096, 0, 5336, 337, 410052, 0, > 0, 0 > vm pgcache: 4096, 0, 3126129, 117,56689945, 24, > 0, 0 > vm pgcache: 4096, 0, 49771, 3910,413657845,1828, > 0, 0 > vm pgcache: 4096, 0, 4249924, 706,224519238, 562, > 0, 0 > % bc > >>> 5336+3126129+49771+4249924 > 7431160 > >>> 7431160*4096/1024/1024/1024 > 28 > >>> > > Look at the graph: > https://imgur.com/yhqK1p8.png > > -- > WBR, Andrey V. Elsukov >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQlTDhvqckfqAKUI>