Date: Tue, 18 May 2021 15:07:44 -0600 From: Alan Somers <asomers@freebsd.org> To: FreeBSD Hackers <freebsd-hackers@freebsd.org>, Mark Johnston <markj@freebsd.org> Subject: The pagedaemon evicts ARC before scanning the inactive page list Message-ID: <CAOtMX2gvkrYS0zYYYtjD%2BAaqv62MzFYFhWPHjLDGXA1=H7LfCg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
I'm using ZFS on servers with tons of RAM and running FreeBSD
12.2-RELEASE. Sometimes they get into a pathological situation where most
of that RAM sits unused. For example, right now one of them has:
2 GB Active
529 GB Inactive
16 GB Free
99 GB ARC total
469 GB ARC max
86 GB ARC target
When a server gets into this situation, it stays there for days, with the
ARC target barely budging. All that inactive memory never gets reclaimed
and put to a good use. Frequently the server never recovers until a reboot.
I have a theory for what's going on. Ever since r334508^ the pagedaemon
sends the vm_lowmem event _before_ it scans the inactive page list. If the
ARC frees enough memory, then vm_pageout_scan_inactive won't need to free
any. Is that order really correct? For reference, here's the relevant
code, from vm_pageout_worker:
shortage = pidctrl_daemon(&vmd->vmd_pid, vmd->vmd_free_count);
if (shortage > 0) {
ofree = vmd->vmd_free_count;
if (vm_pageout_lowmem() && vmd->vmd_free_count > ofree)
shortage -= min(vmd->vmd_free_count - ofree,
(u_int)shortage);
target_met = vm_pageout_scan_inactive(vmd, shortage,
&addl_shortage);
} else
addl_shortage = 0
Raising vfs.zfs.arc_min seems to workaround the problem. But ideally that
wouldn't be necessary.
-Alan
^ https://svnweb.freebsd.org/base?view=revision&revision=334508
[-- Attachment #2 --]
<div dir="ltr"><div>I'm using ZFS on servers with tons of RAM and running FreeBSD 12.2-RELEASE. Sometimes they get into a pathological situation where most of that RAM sits unused. For example, right now one of them has:</div><div><br></div><div><span style="font-family:monospace">2 GB Active</span></div><div><span style="font-family:monospace">529 GB Inactive</span></div><div><span style="font-family:monospace">16 GB Free</span></div><div><span style="font-family:monospace">99 GB ARC total<br></span></div><div><span style="font-family:monospace">469 GB ARC max</span></div><div><span style="font-family:monospace">86 GB ARC target<br></span></div><div><br></div><div>When a server gets into this situation, it stays there for days, with the ARC target barely budging. All that inactive memory never gets reclaimed and put to a good use. Frequently the server never recovers until a reboot.<br></div><div><br></div><div>I have a theory for what's going on. Ever since r334508^ the pagedaemon sends the vm_lowmem event _before_ it scans the inactive page list. If the ARC frees enough memory, then vm_pageout_scan_inactive won't need to free any. Is that order really correct? For reference, here's the relevant code, from vm_pageout_worker:</div><div><br></div><div> <span style="font-family:monospace">shortage = pidctrl_daemon(&vmd->vmd_pid, vmd->vmd_free_count);<br> if (shortage > 0) {<br> ofree = vmd->vmd_free_count;<br> if (vm_pageout_lowmem() && vmd->vmd_free_count > ofree)<br> shortage -= min(vmd->vmd_free_count - ofree,<br> (u_int)shortage);<br> target_met = vm_pageout_scan_inactive(vmd, shortage,<br> &addl_shortage);<br> } else<br> addl_shortage = 0</span></div><div><br></div><div>Raising vfs.zfs.arc_min seems to workaround the problem. But ideally that wouldn't be necessary.</div><div><br></div><div>-Alan<br></div><div><br></div><div>^ <a href="https://svnweb.freebsd.org/base?view=revision&revision=334508">https://svnweb.freebsd.org/base?view=revision&revision=334508</a></div></div>;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2gvkrYS0zYYYtjD%2BAaqv62MzFYFhWPHjLDGXA1=H7LfCg>