Date: Thu, 21 Apr 2022 09:49:02 -0700 From: Doug Ambrisko <ambrisko@ambrisko.com> To: Alexander Leidinger <Alexander@leidinger.net> Cc: Mateusz Guzik <mjguzik@gmail.com>, freebsd-current@freebsd.org Subject: Re: nullfs and ZFS issues Message-ID: <YmGK/hL6OsJFtFzb@ambrisko.com> In-Reply-To: <20220421154402.Horde.I6m2Om_fxqMtDMUqpiZAxtP@webmail.leidinger.net> References: <Yl31Frx6HyLVl4tE@ambrisko.com> <20220420113944.Horde.5qBL80-ikDLIWDIFVJ4VgzX@webmail.leidinger.net> <YmAy0ZNZv9Cqs7X%2B@ambrisko.com> <20220421083310.Horde.r7YT8777_AvGU_6GO1cC90G@webmail.leidinger.net> <CAGudoHEyCK4kWuJybD4jzCHbGAw46CQkPx_yrPpmRJg3m10sdQ@mail.gmail.com> <20220421154402.Horde.I6m2Om_fxqMtDMUqpiZAxtP@webmail.leidinger.net>
index | next in thread | previous in thread | raw e-mail
On Thu, Apr 21, 2022 at 03:44:02PM +0200, Alexander Leidinger wrote: | Quoting Mateusz Guzik <mjguzik@gmail.com> (from Thu, 21 Apr 2022 | 14:50:42 +0200): | | > On 4/21/22, Alexander Leidinger <Alexander@leidinger.net> wrote: | >> I tried nocache on a system with a lot of jails which use nullfs, | >> which showed very slow behavior in the daily periodic runs (12h runs | >> in the night after boot, 24h or more in subsequent nights). Now the | >> first nightly run after boot was finished after 4h. | >> | >> What is the benefit of not disabling the cache in nullfs? I would | >> expect zfs (or ufs) to cache the (meta)data anyway. | >> | > | > does the poor performance show up with | > https://people.freebsd.org/~mjg/vnlru_free_pick.diff ? | | I would like to have all the 22 jails run the periodic scripts a | second night in a row before trying this. | | > if the long runs are still there, can you get some profiling from it? | > sysctl -a before and after would be a start. | > | > My guess is that you are the vnode limit and bumping into the 1 second sleep. | | That would explain the behavior I see since I added the last jail | which seems to have crossed a threshold which triggers the slow | behavior. | | Current status (with the 112 nullfs mounts with nocache): | kern.maxvnodes: 10485760 | kern.numvnodes: 3791064 | kern.freevnodes: 3613694 | kern.cache.stats.heldvnodes: 151707 | kern.vnodes_created: 260288639 | | The maxvnodes value is already increased by 10 times compared to the | default value on this system. With the patch, you shouldn't mount with nocache! However, you might want to tune: vfs.zfs.arc.meta_prune vfs.zfs.arc.meta_adjust_restarts Since the code on restart will increment the prune amount by vfs.zfs.arc.meta_prune and submit that amount to the vnode reclaim code. So then it will end up reclaiming a lot of vnodes. The defaults of 10000 * 4096 and submitting it each loop can most of the cache to be freed. With relative small values of them, then the cache didn't shrink to much. Doug A.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YmGK/hL6OsJFtFzb>