Date: Tue, 18 May 2021 16:37:22 -0500 From: Kevin Day <toasty@dragondata.com> To: Alan Somers <asomers@freebsd.org> Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>, Mark Johnston <markj@freebsd.org> Subject: Re: The pagedaemon evicts ARC before scanning the inactive page list Message-ID: <3CF9B306-6006-41F8-A880-0AE3AF240BF6@dragondata.com> In-Reply-To: <CAOtMX2gvkrYS0zYYYtjD%2BAaqv62MzFYFhWPHjLDGXA1=H7LfCg@mail.gmail.com> References: <CAOtMX2gvkrYS0zYYYtjD%2BAaqv62MzFYFhWPHjLDGXA1=H7LfCg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_797C748B-D663-4CAA-9C40-1EEB701A5B17
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
I'm not sure if this is the exact same thing, but I believe I'm seeing =
similar in 12.2-RELEASE as well.
Mem: 5628M Active, 4043M Inact, 8879M Laundry, 12G Wired, 1152M Buf, =
948M Free
ARC: 8229M Total, 1010M MFU, 6846M MRU, 26M Anon, 32M Header, 315M Other
7350M Compressed, 9988M Uncompressed, 1.36:1 Ratio
Swap: 2689M Total, 2337M Used, 352M Free, 86% Inuse
Inact will keep growing, then it will exhaust all swap to the point it's =
complaining (swap_pager_getswapspace(xx): failed), and never recover =
until it reboots. ARC will keep shrinking and growing, but inactive =
grows forever. While it hasn't hit a point it's breaking things since =
the last reboot, on a bigger server (below) I can watch Inactive slowly =
grow and never free until it's swapping so badly I have to reboot.
Mem: 9648M Active, 604G Inact, 22G Laundry, 934G Wired, 1503M Buf, 415G =
Free
> On May 18, 2021, at 4:07 PM, Alan Somers <asomers@freebsd.org> wrote:
>=20
> I'm using ZFS on servers with tons of RAM and running FreeBSD =
12.2-RELEASE. Sometimes they get into a pathological situation where =
most of that RAM sits unused. For example, right now one of them has:
>=20
> 2 GB Active
> 529 GB Inactive
> 16 GB Free
> 99 GB ARC total
> 469 GB ARC max
> 86 GB ARC target
>=20
> When a server gets into this situation, it stays there for days, with =
the ARC target barely budging. All that inactive memory never gets =
reclaimed and put to a good use. Frequently the server never recovers =
until a reboot.
>=20
> I have a theory for what's going on. Ever since r334508^ the =
pagedaemon sends the vm_lowmem event _before_ it scans the inactive page =
list. If the ARC frees enough memory, then vm_pageout_scan_inactive =
won't need to free any. Is that order really correct? For reference, =
here's the relevant code, from vm_pageout_worker:
>=20
> shortage =3D pidctrl_daemon(&vmd->vmd_pid, vmd->vmd_free_count);
> if (shortage > 0) {
> ofree =3D vmd->vmd_free_count;
> if (vm_pageout_lowmem() && vmd->vmd_free_count > ofree)
> shortage -=3D min(vmd->vmd_free_count - ofree,
> (u_int)shortage);
> target_met =3D vm_pageout_scan_inactive(vmd, shortage,
> &addl_shortage);
> } else
> addl_shortage =3D 0
>=20
> Raising vfs.zfs.arc_min seems to workaround the problem. But ideally =
that wouldn't be necessary.
>=20
> -Alan
>=20
> ^ https://svnweb.freebsd.org/base?view=3Drevision&revision=3D334508 =
<https://svnweb.freebsd.org/base?view=3Drevision&revision=3D334508>;
--Apple-Mail=_797C748B-D663-4CAA-9C40-1EEB701A5B17
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=us-ascii
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">I'm =
not sure if this is the exact same thing, but I believe I'm seeing =
similar in 12.2-RELEASE as well.<div class=3D""><div class=3D""><br =
class=3D""></div><div class=3D"">Mem: 5628M Active, 4043M Inact, 8879M =
Laundry, 12G Wired, 1152M Buf, 948M Free</div><div class=3D"">ARC: 8229M =
Total, 1010M MFU, 6846M MRU, 26M Anon, 32M Header, 315M Other</div><div =
class=3D""> 7350M Compressed, 9988M Uncompressed, =
1.36:1 Ratio</div><div class=3D"">Swap: 2689M Total, 2337M Used, 352M =
Free, 86% Inuse</div></div><div class=3D""><br class=3D""></div><div =
class=3D"">Inact will keep growing, then it will exhaust all swap to the =
point it's complaining (swap_pager_getswapspace(xx): failed), and never =
recover until it reboots. ARC will keep shrinking and growing, but =
inactive grows forever. While it hasn't hit a point it's breaking things =
since the last reboot, on a bigger server (below) I can watch Inactive =
slowly grow and never free until it's swapping so badly I have to =
reboot.</div><div class=3D""><br class=3D""></div><div class=3D"">Mem: =
9648M Active, 604G Inact, 22G Laundry, 934G Wired, 1503M Buf, 415G =
Free</div><div class=3D""><br class=3D""></div><div class=3D""><br =
class=3D""></div><div class=3D""><br class=3D""><div><br =
class=3D""><blockquote type=3D"cite" class=3D""><div class=3D"">On May =
18, 2021, at 4:07 PM, Alan Somers <<a =
href=3D"mailto:asomers@freebsd.org" class=3D"">asomers@freebsd.org</a>>=
wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div =
dir=3D"ltr" class=3D""><div class=3D"">I'm using ZFS on servers with =
tons of RAM and running FreeBSD 12.2-RELEASE. Sometimes they get =
into a pathological situation where most of that RAM sits unused. =
For example, right now one of them has:</div><div class=3D""><br =
class=3D""></div><div class=3D""><span style=3D"font-family:monospace" =
class=3D"">2 GB Active</span></div><div class=3D""><span =
style=3D"font-family:monospace" class=3D"">529 GB =
Inactive</span></div><div class=3D""><span style=3D"font-family:monospace"=
class=3D"">16 GB Free</span></div><div class=3D""><span =
style=3D"font-family:monospace" class=3D"">99 GB ARC total<br =
class=3D""></span></div><div class=3D""><span =
style=3D"font-family:monospace" class=3D"">469 GB ARC =
max</span></div><div class=3D""><span style=3D"font-family:monospace" =
class=3D"">86 GB ARC target<br class=3D""></span></div><div =
class=3D""><br class=3D""></div><div class=3D"">When a server gets into =
this situation, it stays there for days, with the ARC target barely =
budging. All that inactive memory never gets reclaimed and put to =
a good use. Frequently the server never recovers until a =
reboot.<br class=3D""></div><div class=3D""><br class=3D""></div><div =
class=3D"">I have a theory for what's going on. Ever since =
r334508^ the pagedaemon sends the vm_lowmem event _before_ it scans the =
inactive page list. If the ARC frees enough memory, then =
vm_pageout_scan_inactive won't need to free any. Is that order =
really correct? For reference, here's the relevant code, from =
vm_pageout_worker:</div><div class=3D""><br class=3D""></div><div =
class=3D""> <span style=3D"font-family:monospace" =
class=3D"">shortage =3D pidctrl_daemon(&vmd->vmd_pid, =
vmd->vmd_free_count);<br class=3D""> if (shortage > 0) =
{<br class=3D""> ofree =3D =
vmd->vmd_free_count;<br =
class=3D""> if =
(vm_pageout_lowmem() && vmd->vmd_free_count > ofree)<br =
class=3D""> &nb=
sp; shortage -=3D min(vmd->vmd_free_count - =
ofree,<br class=3D""> =
(u_int)shortage);<br =
class=3D""> target_met =3D =
vm_pageout_scan_inactive(vmd, shortage,<br class=3D""> =
&addl_shortage);<br class=3D""> =
} else<br class=3D""> =
addl_shortage =3D 0</span></div><div class=3D""><br class=3D""></div><div =
class=3D"">Raising vfs.zfs.arc_min seems to workaround the =
problem. But ideally that wouldn't be necessary.</div><div =
class=3D""><br class=3D""></div><div class=3D"">-Alan<br =
class=3D""></div><div class=3D""><br class=3D""></div><div class=3D"">^ =
<a =
href=3D"https://svnweb.freebsd.org/base?view=3Drevision&revision=3D334=
508" =
class=3D"">https://svnweb.freebsd.org/base?view=3Drevision&revision=3D=
334508</a></div></div>
</div></blockquote></div><br class=3D""></div></body></html>=
--Apple-Mail=_797C748B-D663-4CAA-9C40-1EEB701A5B17--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3CF9B306-6006-41F8-A880-0AE3AF240BF6>