Date: Tue, 18 May 2021 15:07:44 -0600 From: Alan Somers <asomers@freebsd.org> To: FreeBSD Hackers <freebsd-hackers@freebsd.org>, Mark Johnston <markj@freebsd.org> Subject: The pagedaemon evicts ARC before scanning the inactive page list Message-ID: <CAOtMX2gvkrYS0zYYYtjD%2BAaqv62MzFYFhWPHjLDGXA1=H7LfCg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--00000000000097eef105c2a11a6b
Content-Type: text/plain; charset="UTF-8"
I'm using ZFS on servers with tons of RAM and running FreeBSD
12.2-RELEASE. Sometimes they get into a pathological situation where most
of that RAM sits unused. For example, right now one of them has:
2 GB Active
529 GB Inactive
16 GB Free
99 GB ARC total
469 GB ARC max
86 GB ARC target
When a server gets into this situation, it stays there for days, with the
ARC target barely budging. All that inactive memory never gets reclaimed
and put to a good use. Frequently the server never recovers until a reboot.
I have a theory for what's going on. Ever since r334508^ the pagedaemon
sends the vm_lowmem event _before_ it scans the inactive page list. If the
ARC frees enough memory, then vm_pageout_scan_inactive won't need to free
any. Is that order really correct? For reference, here's the relevant
code, from vm_pageout_worker:
shortage = pidctrl_daemon(&vmd->vmd_pid, vmd->vmd_free_count);
if (shortage > 0) {
ofree = vmd->vmd_free_count;
if (vm_pageout_lowmem() && vmd->vmd_free_count > ofree)
shortage -= min(vmd->vmd_free_count - ofree,
(u_int)shortage);
target_met = vm_pageout_scan_inactive(vmd, shortage,
&addl_shortage);
} else
addl_shortage = 0
Raising vfs.zfs.arc_min seems to workaround the problem. But ideally that
wouldn't be necessary.
-Alan
^ https://svnweb.freebsd.org/base?view=revision&revision=334508
--00000000000097eef105c2a11a6b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>I'm using ZFS on servers with tons of RAM and run=
ning FreeBSD 12.2-RELEASE.=C2=A0 Sometimes they get into a pathological sit=
uation where most of that RAM sits unused.=C2=A0 For example, right now one=
of them has:</div><div><br></div><div><span style=3D"font-family:monospace=
">2 GB=C2=A0=C2=A0 Active</span></div><div><span style=3D"font-family:monos=
pace">529 GB Inactive</span></div><div><span style=3D"font-family:monospace=
">16 GB=C2=A0 Free</span></div><div><span style=3D"font-family:monospace">9=
9 GB=C2=A0 ARC total<br></span></div><div><span style=3D"font-family:monosp=
ace">469 GB ARC max</span></div><div><span style=3D"font-family:monospace">=
86 GB=C2=A0 ARC target<br></span></div><div><br></div><div>When a server ge=
ts into this situation, it stays there for days, with the ARC target barely=
budging.=C2=A0 All that inactive memory never gets reclaimed and put to a =
good use.=C2=A0 Frequently the server never recovers until a reboot.<br></d=
iv><div><br></div><div>I have a theory for what's going on.=C2=A0 Ever =
since r334508^ the pagedaemon sends the vm_lowmem event _before_ it scans t=
he inactive page list.=C2=A0 If the ARC frees enough memory, then vm_pageou=
t_scan_inactive won't need to free any.=C2=A0 Is that order really corr=
ect?=C2=A0 For reference, here's the relevant code, from vm_pageout_wor=
ker:</div><div><br></div><div> <span style=3D"font-family:monospace">short=
age =3D pidctrl_daemon(&vmd->vmd_pid, vmd->vmd_free_count);<br> =
if (shortage > 0) {<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ofree =
=3D vmd->vmd_free_count;<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 i=
f (vm_pageout_lowmem() && vmd->vmd_free_count > ofree)<br>=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 shortage -=3D min(vmd->vmd_free_count - ofree,<br>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (u_int)short=
age);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 target_met =3D vm_pageo=
ut_scan_inactive(vmd, shortage,<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 &addl_shortage);<br> } else<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 addl_shortage =3D 0</span></div><div><br></div><div>Raising vfs.z=
fs.arc_min seems to workaround the problem.=C2=A0 But ideally that wouldn&#=
39;t be necessary.</div><div><br></div><div>-Alan<br></div><div><br></div><=
div>^ <a href=3D"https://svnweb.freebsd.org/base?view=3Drevision&revisi=
on=3D334508">https://svnweb.freebsd.org/base?view=3Drevision&revision=
=3D334508</a></div></div>
--00000000000097eef105c2a11a6b--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2gvkrYS0zYYYtjD%2BAaqv62MzFYFhWPHjLDGXA1=H7LfCg>