Date: Sun, 20 Aug 2023 19:10:12 +0200 From: Mateusz Guzik <mjguzik@gmail.com> To: Alexander Leidinger <Alexander@leidinger.net> Cc: current@freebsd.org Subject: Re: Speed improvements in ZFS Message-ID: <CAGudoHE7RPcHpQEqKbzRM8cJcYKue17=iPVv8iOfZq03h22tTA@mail.gmail.com> In-Reply-To: <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net> References: <61ca9df1b15c0e5477ff51196d0ec073@Leidinger.net> <CAGudoHG5Fgg4184SsXhzqYRR7VPaBXZoirGvyRyJX5ihX5YG-A@mail.gmail.com> <ed1f82dd26d3cc9ec9cc16505109ec40@Leidinger.net> <CAGudoHEP8TrSzz0TL-PsOx0WNc7z3042wJk-jhhVwhTyJ0VEQQ@mail.gmail.com> <88e837aeb5a65c1f001de2077fb7bcbd@Leidinger.net> <4d60bd12b482e020fd4b186a9ec1a250@Leidinger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8/18/23, Alexander Leidinger <Alexander@leidinger.net> wrote: > Am 2023-08-16 18:48, schrieb Alexander Leidinger: >> Am 2023-08-15 23:29, schrieb Mateusz Guzik: >>> On 8/15/23, Alexander Leidinger <Alexander@leidinger.net> wrote: >>>> Am 2023-08-15 14:41, schrieb Mateusz Guzik: >>>> >>>>> With this in mind can you provide: sysctl kern.maxvnodes >>>>> vfs.wantfreevnodes vfs.freevnodes vfs.vnodes_created vfs.numvnodes >>>>> vfs.recycles_free vfs.recycles >>>> >>>> After a reboot: >>>> kern.maxvnodes: 10485760 >>>> vfs.wantfreevnodes: 2621440 >>>> vfs.freevnodes: 24696 >>>> vfs.vnodes_created: 1658162 >>>> vfs.numvnodes: 173937 >>>> vfs.recycles_free: 0 >>>> vfs.recycles: 0 >> >> New values after one rund of periodic: >> kern.maxvnodes: 10485760 >> vfs.wantfreevnodes: 2621440 >> vfs.freevnodes: 356202 >> vfs.vnodes_created: 427696288 >> vfs.numvnodes: 532620 >> vfs.recycles_free: 20213257 >> vfs.recycles: 0 > > And after the second round which only took 7h this night: > kern.maxvnodes: 10485760 > vfs.wantfreevnodes: 2621440 > vfs.freevnodes: 3071754 > vfs.vnodes_created: 1275963316 > vfs.numvnodes: 3414906 > vfs.recycles_free: 58411371 > vfs.recycles: 0 > >>>>> Meanwhile if there is tons of recycles, you can damage control by >>>>> bumping kern.maxvnodes. >> >> What's the difference between recycles and recycles_free? Does the >> above count as bumping the maxvnodes? > > ^^^^^ > >>>> Looks like there are not much free directly after the reboot. I will >>>> check the values tomorrow after the periodic run again and maybe >>>> increase by 10 or 100 so see if it makes a difference. >>>> >>>>> If this is not the problem you can use dtrace to figure it out. >>>> >>>> dtrace-count on vnlru_read_freevnodes() and vnlru_free_locked()? Or >>>> something else? >>>> >>> >>> I mean checking where find is spending time instead of speculating. >>> >>> There is no productized way to do it so to speak, but the following >>> crapper should be good enough: >> [script] >> >> I will let it run this night. > > I have a 51MB text file, compressed to about 1MB. Are you interested to > get it? > Your problem is not the vnode limit, but nullfs. https://people.freebsd.org/~mjg/netchild-periodic-find.svg First, some of the contention is notorious VI_LOCK in order to do anything. But more importantly the mind-boggling off-cpu time comes from exclusive locking which should not be there to begin with -- as in that xlock in stat should be a slock. Maybe I'm going to look into it later. -- Mateusz Guzik <mjguzik gmail.com>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGudoHE7RPcHpQEqKbzRM8cJcYKue17=iPVv8iOfZq03h22tTA>