From nobody Fri Apr 22 14:36:22 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6A961198ECD9 for ; Fri, 22 Apr 2022 14:36:26 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail2.ambrisko.com (mail2.ambrisko.com [70.91.206.91]) by mx1.freebsd.org (Postfix) with ESMTP id 4KlH612BGjz3qCK for ; Fri, 22 Apr 2022 14:36:25 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) IronPort-SDR: H8BMEIoPYf7crTlgGhb/nSdzcmpmDYRKbXp2GqEHIaptiD9USKExHfrrcfAOS4Vm0uKIyVr2xv RZkD0ixZTCtJfWLnPWEdcx8KF4hcP2Osc= X-Ambrisko-Me: Yes IronPort-Data: A9a23:8BdkdqIsP5srdl2FFE+RxZUlxSXFcZb7ZxGr2PjKsXjdYENS0WZVz zZKXGCOa62PMTfwKdxxb4vg8kgB68XTnNA2TQU5pCpnJ55oRWopJjg4wmPYZX76wvUuwCuL1 u1GAjX6BJlcokL0/X9BDJCw9BGQ6onYHtIQOMacUsxAbVcMpBUJ0HqPqMZl6mJcuuVVNivW0 T/ET20zD3f+s9J8Gjp8B6tuM3qDttyq0N8TlgRWifymIDYyPpTIZa/zK51dL1OgKmVVNu+8W +vZyri9uGrc9Q0sEdCi1L38dyXmQJaLbFLI0yQGHfHk2HCupQRquko/HPMZY11WkDaOt9l0w s9Mrp+3DwwuO8UgncxACkIBS34W0apuveWvzWKEmeWXwl3PdXfh2d1qAUA6PIsX9/wxB2xSn dQdKj8QfBGAr+2zybO/DOJrg6wLItPmMYkEtjRr0CvDAPA6aZ7ZTqjA/tMe2y0/7v2it962i 9Excjd1chnaOVtGP10NCYk9m6GjgXyXTtGRk3rNzYJf3oQZ5FUZPGHFPIWHd9qUa99Sm0rE9 GvK836jW0MTMdaFyCGG9Vqlg+XVnDj4X8QZE7jhrqxmh1iax2oyDhwKVAvm+aDo1hbmA98Pe VYJ/icOrLQp8BD5RNfKQBDl8mWPuQQRWoQMHrRiuh2N0Kfd/y2QGnMAEmxacNUjucJvHW4q2 1aFksnHHztqtLHJG3uR+q3O9GG7PCIPLHQBYgcNSAEf4sLgp8c4iReWFoRvF6u8j9vUHzDsw mDX9HFv2+1L1cNSjve151HKhT6ot6PldA9t61WFRH+h4yN4eJWhO96i52/E4KsSN42eVFSA4 iQJwpDM8OAUAJiRvyWRW+FRTqqx7vOIPTCA015iG54tq2ak93K5J9kC4TdiKV1vO8JCcDrje k7IugQX75hWZSP4YahyaoO3KsIr0amwSIy8B6yMNoJDMspraQuK3CByfkrBjWninX8lnbw7J ZrGI92nCmwXCPg/wTfqFf0R16QnmnI3yW/JH8ip1Bm9z7eEPjicTL0fMUCNaaYy66bd+FfZ9 NNWNs2rzRRDUb2jOnCGrdZLdV1af2ImAZ3WqtBMcr/RKwVrL2gtFvvNzO5zYIdihalUyr/F8 3zVtpW0E7Yjaakr8Tm3V00= IronPort-HdrOrdr: A9a23:M6onNKhvFpNm3eGscw7o01aGI3BQXtoji2hC6mlwRA09TyVXra GTddAgpHjJYVcqKRUdcL+7VJVoLUmyyXcx2/h2AV7AZniChILLFvAA0WKK+VSJcEDDH6xmpM VdmsNFaOEYY2IVsS7LijPTL+od Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport2.ambrisko.com with ESMTP; 22 Apr 2022 06:32:52 -0700 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.17.1/8.17.1) with ESMTPS id 23MEaNEY061946 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 22 Apr 2022 07:36:23 -0700 (PDT) (envelope-from ambrisko@ambrisko.com) X-Authentication-Warning: internal.ambrisko.com: Host localhost [127.0.0.1] claimed to be ambrisko.com Received: (from ambrisko@localhost) by ambrisko.com (8.17.1/8.17.1/Submit) id 23MEaMMB061945; Fri, 22 Apr 2022 07:36:22 -0700 (PDT) (envelope-from ambrisko) Date: Fri, 22 Apr 2022 07:36:22 -0700 From: Doug Ambrisko To: Alexander Leidinger Cc: Mateusz Guzik , freebsd-current@freebsd.org Subject: Re: nullfs and ZFS issues Message-ID: References: <20220420113944.Horde.5qBL80-ikDLIWDIFVJ4VgzX@webmail.leidinger.net> <20220421083310.Horde.r7YT8777_AvGU_6GO1cC90G@webmail.leidinger.net> <20220421154402.Horde.I6m2Om_fxqMtDMUqpiZAxtP@webmail.leidinger.net> <20220422090439.Horde.TabULDW9aIeaNLxngZxdvvN@webmail.leidinger.net> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220422090439.Horde.TabULDW9aIeaNLxngZxdvvN@webmail.leidinger.net> X-Rspamd-Queue-Id: 4KlH612BGjz3qCK X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of ambrisko@ambrisko.com has no SPF policy when checking 70.91.206.91) smtp.mailfrom=ambrisko@ambrisko.com X-Spamd-Result: default: False [-1.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; FREEFALL_USER(0.00)[ambrisko]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; DMARC_NA(0.00)[ambrisko.com]; AUTH_NA(1.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.90)[-0.901]; MLMMJ_DEST(0.00)[freebsd-current]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7922, ipnet:70.88.0.0/14, country:US]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Fri, Apr 22, 2022 at 09:04:39AM +0200, Alexander Leidinger wrote: | Quoting Doug Ambrisko (from Thu, 21 Apr 2022 | 09:38:35 -0700): | | > On Thu, Apr 21, 2022 at 03:44:02PM +0200, Alexander Leidinger wrote: | > | Quoting Mateusz Guzik (from Thu, 21 Apr 2022 | > | 14:50:42 +0200): | > | | > | > On 4/21/22, Alexander Leidinger wrote: | > | >> I tried nocache on a system with a lot of jails which use nullfs, | > | >> which showed very slow behavior in the daily periodic runs (12h runs | > | >> in the night after boot, 24h or more in subsequent nights). Now the | > | >> first nightly run after boot was finished after 4h. | > | >> | > | >> What is the benefit of not disabling the cache in nullfs? I would | > | >> expect zfs (or ufs) to cache the (meta)data anyway. | > | >> | > | > | > | > does the poor performance show up with | > | > https://people.freebsd.org/~mjg/vnlru_free_pick.diff ? | > | | > | I would like to have all the 22 jails run the periodic scripts a | > | second night in a row before trying this. | > | | > | > if the long runs are still there, can you get some profiling from it? | > | > sysctl -a before and after would be a start. | > | > | > | > My guess is that you are the vnode limit and bumping into the 1 | > second sleep. | > | | > | That would explain the behavior I see since I added the last jail | > | which seems to have crossed a threshold which triggers the slow | > | behavior. | > | | > | Current status (with the 112 nullfs mounts with nocache): | > | kern.maxvnodes: 10485760 | > | kern.numvnodes: 3791064 | > | kern.freevnodes: 3613694 | > | kern.cache.stats.heldvnodes: 151707 | > | kern.vnodes_created: 260288639 | > | | > | The maxvnodes value is already increased by 10 times compared to the | > | default value on this system. | > | > I've attached mount.patch that when doing mount -v should | > show the vnode usage per filesystem. Note that the problem I was | > running into was after some operations arc_prune and arc_evict would | > consume 100% of 2 cores and make ZFS really slow. If you are not | > running into that issue then nocache etc. shouldn't be needed. | | I don't run into this issue, but I have a huge perf difference when | using nocache in the nightly periodic runs. 4h instead of 12-24h (22 | jails on this system). I wouldn't do the nocache then! It would be good to see what Mateusz patch does without nocache for your env. | > On my laptop I set ARC to 1G since I don't use swap and in the past | > ARC would consume to much memory and things would die. When the | > nullfs holds a bunch of vnodes then ZFS couldn't release them. | > | > FYI, on my laptop with nocache and limited vnodes I haven't run | > into this problem. I haven't tried the patch to let ZFS free | > it's and nullfs vnodes on my laptop. I have only tried it via | | I have this patch and your mount patch installed now, without nocache | and reduced arc reclaim settings (100, 1). I will check the runtime | for the next 2 days. | | Your mount patch to show the per mount vnodes count looks useful, not | only for this particular case. Do you intend to commit it? I should since it doesn't change the size of the structure etc. I need to put it up for review. Thanks, Doug A.