Date: Wed, 04 Mar 2026 11:03:42 +0100 From: Alexander Leidinger <Alexander@Leidinger.net> To: Doug Ambrisko <ambrisko@ambrisko.com>, Mark Johnston <markj@FreeBSD.org> Cc: Rick Macklem <rick.macklem@gmail.com>, Peter Eriksson <pen@lysator.liu.se>, FreeBSD CURRENT <freebsd-current@freebsd.org>, Garrett Wollman <wollman@bimajority.org>, Alexander Motin <mav@freebsd.org> Subject: Re: RFC: How ZFS handles arc memory use Message-ID: <0d466ee1739ff7ddc967d725453dda35@Leidinger.net> In-Reply-To: <aadklpU6TzDi_NJI@ambrisko.com> References: <CAM5tNy5b3=04zC84Q_c60A9qssZTEY2n73okXoFPeT%2BYSK25JQ@mail.gmail.com> <F848B1F3-DE79-49D3-8D1C-1CB1BB2055E3@lysator.liu.se> <aQKB6P3HNKVNQGip@ambrisko.com> <22b478c6bad8212c61ca19a983a8e2e4@Leidinger.net> <aadFxht81oYqaz8h@ambrisko.com> <CAM5tNy4ji=vRhZBBo2JoargVB8vbky_TeamTTC8_i=LHR59Qkw@mail.gmail.com> <aadklpU6TzDi_NJI@ambrisko.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
Am 2026-03-03 23:45, schrieb Doug Ambrisko:
> On Tue, Mar 03, 2026 at 02:25:11PM -0800, Rick Macklem wrote:
> | On Tue, Mar 3, 2026 at 12:33 PM Doug Ambrisko <ambrisko@ambrisko.com>
> wrote:
> | >
> | > On Sun, Nov 02, 2025 at 11:48:06AM +0100, Alexander Leidinger
> wrote:
> | > | Am 2025-10-29 22:06, schrieb Doug Ambrisko:
> | > | > It seems around the switch to OpenZFS I would have arc clean
> task
> | > | > running
> | > | > 100% on a core. I use nullfs on my laptop to map my shared ZFS
> /data
> | > | > partiton into a few vnet instances. Over night or so I would
> get into
> | > | > this issue. I found that I had a bunch of vnodes being held by
> other
> | > | > layers. My solution was to reduce kern.maxvnodes and
> vfs.zfs.arc.max so
> | > | > the ARC cache stayed reasonable without killing other
> applications.
> | > | >
> | > | > That is why a while back I added the vnode count to mount -v so
> that
> | > | > I could see the usage of vnodes for each mount point. I made a
> script
> | > | > to report on things:
> | > |
> | > | Do you see this also with the nullfs mount option "nocache"?
> | >
> | > I seems to have run into this issue with nocache
> | > /data/jail/current/usr/local/etc/cups
> /data/jail/current-other/usr/local/etc/cups nullfs rw,nocache 0 0
> | > /data/jail/current/usr/local/etc/sane.d
> /data/jail/current-other/usr/local/etc/sane.d nullfs rw,nocache 0 0
> | > /data/jail/current/usr/local/www
> /data/jail/current-other/usr/local/www nullfs rw,nocache 0 0
> | > /data/jail/current/usr/local/etc/nginx
> /data/jail/current-other/usr/local/etc/nginx nullfs rw,nocache 0 0
> | > /data/jail/current/tftpboot
> /data/jail/current-other/tftpboot nullfs rw,nocache 0 0
> | > /data/jail/current/usr/local/lib/grub
> /data/jail/current-other/usr/local/lib/grub nullfs rw,nocache 0 0
> | > /data/jail
> /data/jail/current-other/data/jail nullfs rw,nocache 0 0
> | > /data/jail
> /data/jail/current/data/jail nullfs rw,nocache 0 0
> | >
> | > After a while (a couple of months or more). My laptop was running
> slow
> | > with a high load. The perodic find was running slow. arc_prunee
> was
> | > spinning. When I reduced the number of vnodes then things got
> better.
> | > My vfs.zfs.arc_max is 1073741824 so that I have memory for other
> things.
> | >
> | > nocache does help taking longer to get into this situation.
> | Have any of you guys tried increasing vfs.zfs.arc.free_target?
> |
> | If I understand the code correctly, when freemem <
> vfs.zfs.arc.free_target
> | the reaper thread (the one that does uma_zone_reclaim() to return
> pages
> | to the system from the uma keg that the arc uses) should be
> activated.
>
> I haven't tried that. I set:
> kern.maxvnodes
> vfs.zfs.arc.min
> vfs.zfs.arc.max
> vfs.zfs.prefetch.disable=1
>
> I need to make sure kern.maxvnodes is small enough so it doesn't thrash
> when vfs.zfs.arc.max set to 1G. The issues tend to take a while to
> happen. On the plus side I can adjust these when I hit them mostly by
> reducing kern.maxvnodes without having to do a reboot.
There was this commit recently_
https://cgit.freebsd.org/src/commit/sys/fs/nullfs?id=8b64d46fab87af3ae062901312187f3a04ad2d67
I have not checked if this race condition could result in anything
related to what we see. From the commit message I can not deduct if this
could for example lead to a (even temporary) resource leak which may
explain this behavior. Mark, what is the high-level result of this race
condition you fixed in nullfs? At first look at the commit log I would
rather assume vnodes of the lower FS could rather be freed more early
and not at all because of the race condition.
Bye,
Alexander.
--
http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmmoA44ACgkQEg2wmwP4
2IYDKg/9FTVs+hfPa5Tn2eQ2BxhvsUSEELIivdJP/FfF160CAossxcsVZiRU8Y4d
rNSHxaDB58r6SewYQ4ky5yo3zRnJA0+hrgdhow+65u/dY95+op+dELXiSdG72kLU
22+aMGuB5hGXs35Sykhc0hoBAzQRnrynaIg2z4S2L/fO2qI1QObYOTkzRrq98znX
JCY4HY3v1eBVNoHQWzx7fm9ONfSre9K5LDjaGXo/ww65luWiYOWe6o13TMnW1vD5
ioOjM9S7GvXMOFWi6vA05/JyAEb107oy80JDOijR0U/xn2KI1mRHeGskRLE8yaYp
ZfFgTYskcDxfVkzK2wGIu1ue8pUIaVTCWtYjrdGPEtD6R6vdbZgC0JPY4orsfFoU
MWOEx9LgOVqjJEOOGQJzltrQi2f5fS0D7gt3Nz3xNpO7ypmeo/2SDypEvyZVNddw
gkU3AZ+wiH2MXWLtzKHsWUuVXEIjK7G3I7O5CzvoGNs8i9B9N3+uycWQ7PJgo+I6
PRKxaTHqUXqvL9agUIu9NAHfHV8x3hHmYXhnXJG/boGnM66Wcb+jplANt2dqJ/oH
PwpwJEwZFg/tjL0Zeg1rjpgAM+3Kw8ftTUGSBxP819/9ywyB+qtM9Lasp8B2kKqR
9QBho3iqWdgK90owxEACFaNkCk0cjzckWV0zgy89Pe3j1k0UaSM=
=TVeK
-----END PGP SIGNATURE-----
home |
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0d466ee1739ff7ddc967d725453dda35>