Date: Tue, 3 Aug 2021 16:59:34 +0300 From: "Andrey V. Elsukov" <bu7cher@yandex.ru> To: Mark Johnston <markj@freebsd.org> Cc: =?UTF-8?Q?=c3=96zkan_KIRIK?= <ozkan.kirik@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org>, freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: Wired Memory Increasing about 500MBytes per day Message-ID: <35e9249e-37f7-3ff8-23c5-a22d8b909f09@yandex.ru> In-Reply-To: <YQlI4cfsiCgH74WK@nuc> References: <CAAcX-AFWQrRjcniS5gKHzFuw3KKVJ-GKkFGmaA6OLwQ5vu9Jyg@mail.gmail.com> <5dc957ec-9483-0a80-b29e-be4b71c1b9d9@yandex.ru> <YQlI4cfsiCgH74WK@nuc>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --AyljISCoKMiz299w3FED5YVd1ZGjCksFl Content-Type: multipart/mixed; boundary="ONABrLSmCq904Ths9Cl9WpnnbYBsOT961"; protected-headers="v1" From: "Andrey V. Elsukov" <bu7cher@yandex.ru> To: Mark Johnston <markj@freebsd.org> Cc: =?UTF-8?Q?=c3=96zkan_KIRIK?= <ozkan.kirik@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org>, freebsd-stable <freebsd-stable@freebsd.org> Message-ID: <35e9249e-37f7-3ff8-23c5-a22d8b909f09@yandex.ru> Subject: Re: Wired Memory Increasing about 500MBytes per day References: <CAAcX-AFWQrRjcniS5gKHzFuw3KKVJ-GKkFGmaA6OLwQ5vu9Jyg@mail.gmail.com> <5dc957ec-9483-0a80-b29e-be4b71c1b9d9@yandex.ru> <YQlI4cfsiCgH74WK@nuc> In-Reply-To: <YQlI4cfsiCgH74WK@nuc> --ONABrLSmCq904Ths9Cl9WpnnbYBsOT961 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable 03.08.2021 16:47, Mark Johnston =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> We noticed the same problem, I'm not sure the exact version, but you c= an >> check the output: >> # vmstat -z | egrep "ITEM|pgcache" >> >> The page cache grows until lowmem is not reached. Then it automaticall= y >> cleans and begins to grow again. >=20 > The pgcache zones simply provide a per-CPU cache and allocator for > physical page frames. The sizes of the caches are bounded. The number= s > of "used" items from the pgcache zones do not really tell you anything > since those pages may be allocated for any number of purposes, includin= g > for other UMA zones. For instance, if ZFS allocates a buffer page from= > its ABD UMA zone, and that zone's caches are empty, UMA may allocate a > new slab using uma_small_alloc() -> vm_page_alloc() -> pgcache zone. >=20 > So if there is some wired page leak, the pgcache zones are probably not= > directly responsible. We don't see any leaks, but our monitoring shows that "free" memory migrates to "wired" and only these zones are grow. So, we have on the graphs linear growing of wired memory over 7 days. When free memory reaches ~4% all returns to normal, and then again linear growing for 7 days. And pgcache zones reset their number of USED items to low value. This is on the server with 256G RAM. E.g. This is when 9% of free memory left: $ vmstat -z | egrep "ITEM|pgcache" ITEM SIZE LIMIT USED FREE REQ FAILSLEEP XDOMAIN vm pgcache: 4096, 0, 5225, 139, 412976, 0, 0, 0 vm pgcache: 4096, 0,28381269, 77,190108006, 24, 0, 0 vm pgcache: 4096, 0, 166358, 11523,1684567513,3054, 0, 0 vm pgcache: 4096, 0,29548679, 576,780034183,1730, 0, 0 $ bc >>> 5225+28381269+166358+29548679 58101531 >>> 58101531*4096/1024/1024/1024 221 >>> This is when lowmem triggered: % vmstat -z | egrep "ITEM|pgcache" ITEM SIZE LIMIT USED FREE REQ FAILSLEEP XDOMAIN vm pgcache: 4096, 0, 5336, 337, 410052, 0, 0, 0 vm pgcache: 4096, 0, 3126129, 117,56689945, 24, 0, 0 vm pgcache: 4096, 0, 49771, 3910,413657845,1828, 0, 0 vm pgcache: 4096, 0, 4249924, 706,224519238, 562, 0, 0 % bc >>> 5336+3126129+49771+4249924 7431160 >>> 7431160*4096/1024/1024/1024 28 >>> Look at the graph: https://imgur.com/yhqK1p8.png --=20 WBR, Andrey V. Elsukov --ONABrLSmCq904Ths9Cl9WpnnbYBsOT961-- --AyljISCoKMiz299w3FED5YVd1ZGjCksFl Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAmEJS8YFAwAAAAAACgkQAcXqBBDIoXrY Bgf+Ix+x4QzmAJ7KGDeB7icUWNfbXHlnklFTchXMaqAbr0QngrXm2BGhLx8bPX/r8WryvgJGv+xk gk2u6hHJdvjiYoF6pd3GOj2CxL/GM6qdweHI0G1iDz/2EbJnmEbH0gEdDNawqEXUx6+IzogzES+a wLwVweVYeQGMLBxg/5SjzCl/GlSCeaKwVZRnlybhhr2gMRQh6S40bNsSxi6sl1Bbgzhb/XF4eZUG EHlcbKT0/4+m3MW2QC697I7z9Kf96R7YG0K0zjJGkgrjds0fMQsvV1TKgwtAZ9B2dvXK9qyAfLBx RW8RimbrxoUpGq4pM0ceW33iKjroUlnr246zJCm6gQ== =2Ei0 -----END PGP SIGNATURE----- --AyljISCoKMiz299w3FED5YVd1ZGjCksFl--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?35e9249e-37f7-3ff8-23c5-a22d8b909f09>