Date: Mon, 15 Jul 2024 13:41:15 -0700 From: Jim Long <freebsd-questions@umpquanet.com> To: Dan Langille <dan@langille.org> Cc: freebsd-questions@freebsd.org Subject: Re: Unable to limit memory consumption with vfs.zfs.arc_max Message-ID: <ZpWJa%2BVLW1apLA%2BD@sfo.umpquanet.com> In-Reply-To: <ZpWFhbZFukxWyzMP@sfo.umpquanet.com> References: <ZjUYzdiz01GgnbX/@sfo.umpquanet.com> <4299f196-1fbe-4590-8668-d023d7044e8d@sentex.net> <ZjVLxKH5LDBZjakR@sfo.umpquanet.com> <3d2f7913-b91d-4d00-a9b4-f37cf1032fc0@app.fastmail.com> <ZpWFhbZFukxWyzMP@sfo.umpquanet.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--fp5xN3iEGoS62Fc5
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
As Bugs Bunny often said, "What a maroon!"
Here's the attached MRTG graph.
On Mon, Jul 15, 2024 at 01:24:37PM -0700, Jim Long wrote:
> Picking up this old thread since it's still vexing me....
>
> On Sat, May 04, 2024 at 07:56:39AM -0400, Dan Langille wrote:
> >
> > This is from FreeBSD 14 on an Dell R730 in the basement (primary purpose, poudriere, and PostgreSQL, and running four FreshPorts nodes):
> >
> > >From top:
> >
> > ARC: 34G Total, 14G MFU, 9963M MRU, 22M Anon, 1043M Header, 9268M Other
> > 18G Compressed, 41G Uncompressed, 2.28:1 Ratio
> >
> > % grep arc /boot/loader.conf
> > vfs.zfs.arc_max="36000M"
> >
> > Looks like the value to set is:
> >
> > % sysctl -a vfs.zfs.arc | grep max
> > vfs.zfs.arc.max: 37748736000
> >
> > Perhaps not a good example, but this might be more appropriate:
> >
> > % grep vfs.zfs.arc.max /boot/loader.conf
> > vfs.zfs.arc_max="1200M"
> >
> > with top showing:
> >
> > ARC: 1198M Total, 664M MFU, 117M MRU, 3141K Anon, 36M Header, 371M Other
> > 550M Compressed, 1855M Uncompressed, 3.37:1 Ratio
>
> Thank you, Dan, I appreciate you chiming in.
>
> Unfortunately, I think I have those bases covered, although I'm open to
> anything I may have missed:
>
> # grep -i arc /boot/loader.conf /etc/sysctl.conf
> /boot/loader.conf:vfs.zfs.arc.max=4294967296
> /boot/loader.conf:vfs.zfs.arc_max=4294967296
> /boot/loader.conf:vfs.zfs.arc.min=2147483648
> /etc/sysctl.conf:vfs.zfs.arc_max=4294967296
> /etc/sysctl.conf:vfs.zfs.arc.max=4294967296
> /etc/sysctl.conf:vfs.zfs.arc.min=2147483648
>
> # top -b
> last pid: 16257; load averages: 0.80, 1.15, 1.18 up 0+02:03:34 12:05:06
> 55 processes: 2 running, 53 sleeping
> CPU: 11.7% user, 0.0% nice, 18.4% system, 0.1% interrupt, 69.9% idle
> Mem: 32M Active, 141M Inact, 11G Wired, 3958M Free
> ARC: 10G Total, 5143M MFU, 4679M MRU, 2304K Anon, 44M Header, 219M Other
> 421M Compressed, 4744M Uncompressed, 11.28:1 Ratio
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 11057 root 1 127 0 59M 33M CPU0 0 60:16 82.28% ssh
> 11056 root 5 24 0 22M 12M pipewr 3 6:00 6.25% zfs
> 1619 snmpd 1 20 0 34M 14M select 0 0:06 0.00% snmpd
> 1344 root 1 20 0 14M 3884K select 3 0:03 0.00% devd
> 1544 root 1 20 0 13M 2776K select 3 0:01 0.00% syslogd
> 1661 root 1 68 0 22M 9996K select 0 0:01 0.00% sshd
> 1587 ntpd 1 20 0 23M 5876K select 1 0:00 0.00% ntpd
> 14391 root 1 20 0 22M 11M select 3 0:00 0.00% sshd
> 2098 root 1 20 0 24M 11M select 1 0:00 0.00% httpd
> 1904 root 1 20 0 24M 11M select 2 0:00 0.00% httpd
> 1870 root 1 20 0 19M 8688K select 2 0:00 0.00% sendmail
> 2067 root 1 20 0 19M 8688K select 1 0:00 0.00% sendmail
> 2066 65529 1 20 0 13M 4564K select 2 0:00 0.00% mathlm
> 1883 65529 1 20 0 11M 2772K select 3 0:00 0.00% mathlm
> 14397 root 1 20 0 14M 4568K wait 1 0:00 0.00% bash
> 1636 root 1 20 0 13M 2608K nanslp 0 0:00 0.00% cron
> 2082 root 1 20 0 13M 2560K nanslp 3 0:00 0.00% cron
> 1887 root 1 20 0 13M 2568K nanslp 2 0:00 0.00% cron
>
> # sysctl -a | grep m.u_evictable
> kstat.zfs.misc.arcstats.mfu_evictable_metadata: 0
> kstat.zfs.misc.arcstats.mfu_evictable_data: 0
> kstat.zfs.misc.arcstats.mru_evictable_metadata: 0
> kstat.zfs.misc.arcstats.mru_evictable_data: 0
>
> An mrtg graph is attached showing ARC bytes used
> (kstat.zfs.misc.arcstats.size) in green, vs. ARC bytes max
> (vfs.zfs.arc.max) in blue. We can see that daily, the ARC bytes used
> blows right past the 4G limit. Most days, it is brought under control
> by two reboots in /etc/crontab ("shutdown -r now" at 02:55, 05:35),
> although some days the system is too far gone by the time the cron job
> rolls around, and the system stays hung until I can get to the data
> center and power cycle it.
>
> I'm not very skilled at kernel debugging, but is a kernel PR in order?
> This has happened with a GENERIC kernel across at least two builds of
> 14-STABLE:
>
> FreeBSD 14.0-STABLE #0 stable/14-n267062-77205dbc1397: Thu Mar 28 12:12:02 PDT 2024
> FreeBSD 14.1-STABLE #0 stable/14-n267886-4987c12cb878: Thu Jun 6 12:24:06 PDT 2024
>
> Would it help to reproduce this with a -RELEASE version?
>
>
> Thank you again, everyone.
>
> Jim
--fp5xN3iEGoS62Fc5
Content-Type: image/png
Content-Disposition: attachment; filename="arc-month.png"
Content-Transfer-Encoding: base64
iVBORw0KGgoAAAANSUhEUgAAArwAAAGzBAMAAAFILemuAAAAHlBMVEX19fXCwsJkZGT/AAAA
zAAAAAAAZgDvn0//AP8AAP/G7XGyAAAACXBIWXMAAA7EAAAOxAGVKw4bAAARTUlEQVR4nO1d
y5KbyBYkpimt78wXTPgLHNH24u6IcJk1i2YivGPB+IdueOG/vVC8iuIUnHpJIOUZfIzRUZKk
slO0mqGzPx0q+0/mUBimhhvfYesT7zwsizjDIhkyhl9ouBr+1h8+RMbwqwwLc7hwQX7u4dx9
OI883KjhXErJGC6I4W6t5AwX/YZuTa6G+4ePy/esoJqbmJv2yHqYUw5nak64qMNqLjls++p2
HK7MDdGQMfxCw2K76fScMXyv4Y07TsK5OMNw4zJcqGHm9DBcHs6NZ1GykHIZblyGi/V56npY
quH+7Cwvxsn1MOtMjcRnFHlqRm7kVrLzrx5Y+DDiACcpAN8fuLkv8M19h88MPAQZgAEMYAAD
uK/5hCUi8PokqIgHvC4AAxjAZwIuvIC1b+jiAmv1LMD5pYBvTSrgYgHOtw/7A+dpgdWHKTI/
/ljnEHgmp4ClHdhyFJ7A6tFCG9x8tmMANzpwXqhPkjq8DrNUwHLe2K+0PVy/z1z9BDRXKzqw
6E8ueD8eZH8ydZ/v/oWoMjGcGVV1ltUHGzP7pNMPHF0qmRB/JwauiB9hRAGOXwAGMID5wE0a
4O4kuDtHU11fMvpylyjAzYS9bNeet0z6ABcG8O4C4HsDFw8CvvWnsZcCzl8EWCQCrq4nBYAB
DGAAAxjArwBcXQ64rxQnLCKZxpcDFklfPAADGMAABrAPcJGvgId/AvjkwCIRcJVWipxeAAxg
AO8A59cDbhfgFsAzcBkdWLZSXW6iNt7GhxqbQ/eA5QpYXcHSV2xgaQfO0wJLD2BJAKtHixTA
/Yr0Be5wFHDRasAyLwbg7h9tD58PVz0Val7m5QZYxqv1NSxfEgCr+pxFrL8PJyIC8//Hv51J
A1hUVVYNj9RimjY2CtZGinGU+jvdZWlJ6g/wTVo63/nDwqw+Z6351kJlm6jquyrmVlf2wxUK
fNMW+KYt8E1bT8P3Ni2ttj4vDytPvlv6+TS8BeEUV4N78I2yJOPbUiDRKIMv+D4f3zyM7236
6AR8k/DNwRd8H8B3+bykErdpabV125JVIp+GuyX/+C67hfFEp8X+eclF9L2aH8AXfMEXfMEX
fMEXfMEXfMEXfF+Or/kT+bPzvZq+5n2JrsFXu74kU5+BqOVfbf1w+XeYl+oDE5cnchbj85KM
/ryEI9Pd/WtcuHZyvmLPvyfku5sP4Au+4Au+4Au+4Au+4Au+fnxztTwL33xawBd8n4ivvB/f
5fOSX79/nG8xPy/JjM9LTq+v7odc45sfLeALviRfCb7gC77J+f68GN+2uBjf8jn4yuKcfOWz
8JXn59utT3y7lZ8n4Ns2LL5qpdXGbutn3Y5kMPfiy7e/ZYmFr6qJr2TzXY1dge+yl+5Zd+Ar
I/Il9vg4vrN88rF8pYVv//RixVcOCF10FPnCVzlqxXe7lyO+056XfJqRpo3tzLeL2olmq/Od
h2e+g+7jb0Ir82ljO22XM/Fpe3vEV56xChvft/dH36mErg3f17t/CXmHJbL8fvcahy+JHLyR
eBUtk7oG17s/wadrVapfapio/rqcH8A3ZYFv2tL4jrFc8d+iHlA631qR7f4TDucgX1Od3BDI
Bt/h9obqL/LYyK2f2dK4vm4b5B5g4StqdVWwer928ATJN9BSFY285uu5P76+ZO3syUCunPme
yw+rR+AHS8EPrqgTckDBD1bUnYIffPYHP1iR4Qf3gh9cUSfkgIIfrKg7BT8YT7hN/7wRI1Od
xQ/zT23Leb1a/yjXlW9aPxB8yeXzuPIzhBpBzNUPHnyHYb3MF8NGjfReTD848SWfnu1MDgPJ
/bDw1VTzWaoV8nx8afygS8yltos8T6bxg5XvDjUO38h+KHW+Cfywz1c9nnvzfYwfTL67fjjm
m9wP/vrCD4d+yC/mh5mvnLSerxAK8YP59Gh+2Oeba3z1MUPffEPQ5FuZD+nIbD/8Q15vVJmo
OZtvPlHb0XeeHJfP5hHs8K0ovtSi883n6z0/+sttSupVoU21fiivFr764qovuZB85YpvvsO3
Wmu6QV4WHz+E8KWfnrH5xvEDzfeD4mtz1HbjKfxALxbRT+wHLt+z+MEq+r39cPDE0/nB5Ht2
P7D0jeOHjyv4oRo+5es7zfdkfqiy+fqSjzr7ELzla9//MTbW6oZtTITtUi/I+mJeXzLy7f5c
wQ+irqvp+pJL+EH/ekM+RPfDMV/4AX6AH+AH+AF+gB/gB/gBfoAf4Af4AX6AH+AH+AF+gB/g
B/gBfoAf4Af4AX54iB8qHt/T+GFz6x74IaoftvrCDwn8cHR9yWn8MOaDWK4vqUQ+LVJbJ5bP
fb9NY3JYUb8O8OCJh8tnc8t0fYmoq0nf6fqSS/hhuX/Jyf2wuVHayfNh8wZ3DT9c5f1ie9u8
c/uB4As/RPQDl+9Z/EDwhR/gB/gBfoAf4Af4AX6AH+AH+AF+gB/gB/jB2Q/5xfzA4nsiP5h8
4Qf4AX6AH+AH+AF+gB/gh1f0w/BzTnV9ySX8UM3Xl9S/f/xiLv9lT5LLDzbyjx/m7wOsq/37
l5zND9lyfckl/IB84FLz9QPeL+AH+AF+ONT3zn64XcwPtz3kE/qBwxd+gB/gB/jhLn4oLuYH
eTE/cPieyQ+yuJgf1nwXfZuT+sHCVzYmtXP7Qed7BT8c8j2bH4qL+aHYe+4J/XDE92x+kOf3
Q6fpB8X3rH7o+A76qpUz+EF68X2cH3b4diuzHzj63sUPh3yd/UCN3cUPO3z3/MDgeyo/yHYB
L6/gB42vXCGf1A/HfOP6oeT6YT5jK+187+AHyfXDvEU+1A8MvpXBt3ikH9Z8DT80G307V3Qr
zUQ5l+1Mf/bDl4bPd55q7Hy74OmWmW/PrqH17VcWOu24RS7U+uFSbSzHgZ7v8M/puQw/THte
YQ98FXC/rPj2ux3m5UhBjlKuqE2AcpJ1vXFU6Ev3aL/MDtnhO76gjPpCbeQ/nawPGnmHLx+b
5OtKjYP8YeHbX1/yzsfm83UV/VDf8X4gWfX2XrPrK3/UsbbINN/uz3tFmGR726a0GytqI33/
kvMW/fV23trjK/oLvPqLpmpBbKzIjeTTqY366P6O9I37fOtuVu2sojZSk4KaFILa6LWjI77D
L60ydjhsNLD3JqmNOpX9STZf9bXXvW716mWeN/Inw55e66Sf6evtjPVH9ulalf2JSledvI92
5DMX5E1akDdpQd6kRckrphO94cSuO4va3nwOxSpC3u5se5Bz+Lv/szlhRvGKdu8k7NBr7V+x
6409+divIH3vx5znabu8ohq+LReai+MXX94U5XdM+5xXWvHf2lyo8GdTyJva54HutT0F4TBU
InlTFMLBgwrCYSyEg/veEQ7MQjh4FMJhLoQDsYN4hXDwoIJwGAvh4L53hAOzEA4ehXCYC+FA
7CBeIRw8qCAcxkI4uO8d4cAshINHXTkcfmZGb0tzy9hZhXBYP4UvL6cjHAwqLHlpn1P9bVm/
qaahjci2sn6NvEY4RJZ3I5oN86Z1h5jaxbxoOIS7N6TrovlxzhbmUz1XOBCdv3e//nY4c6Fw
mERmh0OSrh97InlpqHvJu6nIonlxtmCeNhzyCPJePxzaBSque+3yvlI4PEDeTT1xOBzIi3AY
C+FwrnBotb6VV9t+inA44HO+cNjKKzeH09I+pw52I6+GxpKXwLTs3a/vc44eDjZ5203fl7fV
qA7rB/JuwmF/XyHyEpj6S/ZmzkSRt9wclOpWeW3dVd4RX7s5yZaJTZaDrx1LONhwxv5m2b5g
JpZ343Nrt8irIbPk3RwgT95Nd+Vs6V7y0qI5u9dT3u9ceZ2/gjjP0o89kbw01L3kZbvXoW/e
BCO499nDwWHvfv0Zw0FGDQc/nz9xOMhk4RDZvdcMBwd5EQ4e8mrhELRHv/5K4RBN3jOFg+Wg
HhAOB/KePBxENt7HbZga70YiJMJhw9lDXjHZe303Ehf38vsLhoMYHzfuRuIgL8JhLNvdSLLt
3UgQDnHkpQvhECkc6EI4JHYvwmHDOaa8NBTCAeEQtSMconeEQ6BoXpwtmAiHgI5wiN4RDoGi
eXG2YCIcAjrCIXpHOASK5sXZgolwCOgIh+gd4RAomhdnCybCIaAjHKJ3hEOgaF6cLZgIh4CO
cIjeEQ6BonlxtmAiHAI6wiF6RzgEiubF2YKJcAjoCIfoHeEQKJoXZwsmwiGgIxyid4RDoGhe
nC2YCIeAjnCI3hEOgaJ5cbZgIhwCOsIhekc4BIrmxdmCiXAI6AiH6B3hECiaF2cLJsIhoPu4
t5o3WQrhECCvEFmV7RbCISQcGO6lAREOnHCos4NCOCAcHPv9wqFe2Ze8GwkNiHDghMPqBq7T
jTKmV2K4GwnCYcOZK2+3QQuHtbzudyPh9xcJh/V5g7oDyeZuJDQgdbAIh7Fmeev68K2Nph3m
pRcJh37Tvr4u7uX3VwmHzrzx5EU4jDXJW4uj7ysQDv7y9sng+W0FwuE4HLp3tuO3NhoQ4cAI
h+NCOCSWl4ZCOByHA6MQDgiHZB3hECiaF2cLJsIhoCMconeEQ6BoXpwtmAiHgI5wiN4RDoGi
eXG2YCIcAjrCIXpHOASK5sXZgolwCOgIh+gd4RAomhdnCybCIaAjHKJ3hEOgaF6cLZgIh4CO
cIjeEQ6BonlxtmAiHAL6NcMhV80qL8IhqbwIBw/aKeRFOHjIi3BIKi/CwYM2wmGsVTjIpTt7
Seod4TDWyr0PkPeVwuFA3l3ayeVFOCAczEI4IByO9x6hIxwQDkNHOIyFcHjKcBjxnyscDu5G
8gB5nykc5js5aHcj+c2vX+zJ/zmgxt+7Xx1z/rUv73DzAfNuJAiHYT1KOOzejQThEBQORImV
LEuXxHaxrOuS3kneK585sOTl97dl/UDepwqHbbm4l9/58iIcqHB4gLwIB4SDWQgHhMN8gAgH
1REOY90/HKSne/l95fMb91lPEg6+8nqGQyR5+3aJcFjLmzwcduVFOCActh3hMBbCAeHg2BEO
CAdzC8IB4aD6S4VDWyAcErq3lQXCwez3khfhEBoOUjYIB6PHdO+evNtwkLJEOOz3EPeGy4tw
8JAX4UDKK53DQSIc9jtfXsq9ofK+VDjEkhfhECscCoTDbufIq97CaPfKBuGw13nyyoaQd3jA
tvd2d7/UozcPzpZ+qXAY5P2+EngSnJJ3QYgnr37sieSloXjytp0eAfIWa/cuW8LkXRnmRjPc
5Wx5Ie4eDtJLXqWfs7y+4dBF/FXDodfDS95iEvM7W94FwVHebme2Z508HMLlTR8OtLxDzls4
nyccirvJ6x0OsqTlNfaCcIgsb/Ek4VA8Nhzs8lo4XywcZOxwkNI8H+i/BXR3r4XzxcKhOz9W
6+xwKGx7XOQ1XgLF90XDQeo/R+O4lyNvMXOYPrtorOGgCFw7HKQ61flpUh3lXb54XeQdZCmX
vWvylquZQd7huaXOMx/lbbQXgpT327B9xCznI1pmIoXDXGWmHTKVh4a84zdjpLw91XYRduiN
NRz6bweJvbckQ7mWV5dlqWwzX66O/dvMv1hNBsprCltqlMp1JyaLlR3LcZ2Ud5zRd6JtWc2U
2gFOex88Kee960Djq9UO276T8uoMp2PMNMxv2+NVvVGPeocDv74fj4z1zQE1/t796pjz9wN5
hdanu5G8p6DNlze1aPy9B8s73ShjMnqdZW/v71+/yS+8hT/5ngAz9eQ+ZzXpIq+6G0knryAm
LUHy1JP7z1o9ar+Xjnk3kuydzQU1Ff+tDeVRnvKK6Wtg/Kt7+6sr5qSoay6m4E46YGZcnvwj
sk36yyvEACtGtkJYImk7aT1AApO9d5sU271X7EmHI7Ix9Ze3M0y/THvYkdeYtHEJwmRPCvuL
641pf7PzlbeqxQhcTXuwkNlMWo4vCNNh7zZ5g/ZuK7y1JS3Im7Q+ZX99QqWr/wNBWbg3nxFN
qQAAAABJRU5ErkJggg==
--fp5xN3iEGoS62Fc5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ZpWJa%2BVLW1apLA%2BD>