From nobody Mon Apr 25 16:44:19 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BC6771A9BAFC for ; Mon, 25 Apr 2022 16:44:22 +0000 (UTC) (envelope-from ltning@anduin.net) Received: from mail.anduin.net (mail.anduin.net [185.42.170.45]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kn9pF3nKGz3mCx for ; Mon, 25 Apr 2022 16:44:21 +0000 (UTC) (envelope-from ltning@anduin.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=anduin.net; s=dkim2021; h=Content-Transfer-Encoding:MIME-Version:Content-Type:References: In-Reply-To:Date:To:From:Subject:Message-ID:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=TM6BhJvhYhljCX0El7Y709H+jfrgGOSwoCtXxtaJuuk=; t=1650905061; x=1651769061; b=BBXW+By+aMPX52c+TKXFezve0Px/bcIXr2HaRGYdAqRET7ijeXGmoEOZaep8+PN6LMc+oISUJzF mHGrFgyfe/+GwESQ2Xp35rj8qq7jQTyKVzwZwfePXuuzNYdldiTsW/7fB3Yk0oOZ+QaHXp+sU/eJm NSPFr2PcdpaPt9FZ4/wLNWl1jGM1O4uw0ZM2wIVelsdjFL3P2CRLuY9u0ofIW3rNPR8lYLWytxluE zYaCZ03Z4bj7KC6hkKi+OOXLbsBdxNsYXeAHjg5fgBA2vUxO83UVdrH8/eNrZXnAxyOJvGQbfFiQV bfq5oy2ezPnFdqhLs/RaodIWwg1lDFhvWlNA==; Received: by mail.modirum.com with esmtpsa (TLS1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.95 (FreeBSD)) (envelope-from ) id 1nj1oy-000EXX-81 for freebsd-current@freebsd.org; Mon, 25 Apr 2022 16:44:20 +0000 Message-ID: Subject: Re: nullfs and ZFS issues From: Eirik =?ISO-8859-1?Q?=D8verby?= To: freebsd-current@freebsd.org Date: Mon, 25 Apr 2022 18:44:19 +0200 In-Reply-To: <20220425152727.Horde.YqhquyTW0ZM3HAbI1kyskic@webmail.leidinger.net> References: <20220420113944.Horde.5qBL80-ikDLIWDIFVJ4VgzX@webmail.leidinger.net> <20220421083310.Horde.r7YT8777_AvGU_6GO1cC90G@webmail.leidinger.net> <20220421154402.Horde.I6m2Om_fxqMtDMUqpiZAxtP@webmail.leidinger.net> <20220422090439.Horde.TabULDW9aIeaNLxngZxdvvN@webmail.leidinger.net> <20220424195817.Horde.W5ApGT13KmR06W2pKA0COxB@webmail.leidinger.net> <20220425152727.Horde.YqhquyTW0ZM3HAbI1kyskic@webmail.leidinger.net> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.42.4 FreeBSD GNOME Team List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-SA-Authenticated: Yes X-Rspamd-Queue-Id: 4Kn9pF3nKGz3mCx X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=anduin.net header.s=dkim2021 header.b=BBXW+By+; dmarc=none; spf=pass (mx1.freebsd.org: domain of ltning@anduin.net designates 185.42.170.45 as permitted sender) smtp.mailfrom=ltning@anduin.net X-Spamd-Result: default: False [-2.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[anduin.net:s=dkim2021]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[anduin.net]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DKIM_TRACE(0.00)[anduin.net:+]; NEURAL_HAM_SHORT(-0.61)[-0.612]; MLMMJ_DEST(0.00)[freebsd-current]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_MIXED_CHARSET(0.71)[subject]; ASN(0.00)[asn:62248, ipnet:185.42.170.0/24, country:EE]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Mon, 2022-04-25 at 15:27 +0200, Alexander Leidinger wrote: > Quoting Alexander Leidinger (from Sun, 24 > Apr 2022 19:58:17 +0200): > > > Quoting Alexander Leidinger (from Fri, 22 > > Apr 2022 09:04:39 +0200): > > > > > Quoting Doug Ambrisko (from Thu, 21 Apr > > > 2022 09:38:35 -0700): > > > > > > I've attached mount.patch that when doing mount -v should > > > > show the vnode usage per filesystem. Note that the problem I was > > > > running into was after some operations arc_prune and arc_evict would > > > > consume 100% of 2 cores and make ZFS really slow. If you are not > > > > running into that issue then nocache etc. shouldn't be needed. > > > > > > I don't run into this issue, but I have a huge perf difference when > > > using nocache in the nightly periodic runs. 4h instead of 12-24h > > > (22 jails on this system). > > > > > > > On my laptop I set ARC to 1G since I don't use swap and in the past > > > > ARC would consume to much memory and things would die. When the > > > > nullfs holds a bunch of vnodes then ZFS couldn't release them. > > > > > > > > FYI, on my laptop with nocache and limited vnodes I haven't run > > > > into this problem. I haven't tried the patch to let ZFS free > > > > it's and nullfs vnodes on my laptop. I have only tried it via > > > > > > I have this patch and your mount patch installed now, without > > > nocache and reduced arc reclaim settings (100, 1). I will check the > > > runtime for the next 2 days. > > > > 9-10h runtime with the above settings (compared to 4h with nocache > > and 12-24h without any patch and without nocache). > > I changed the sysctls back to the defaults and will see in the next > > run (in 7h) what the result is with just the patches. > > And again 9-10h runtime (I've seen a lot of the find processes in the > periodic daily run of those 22 jails in the state "*vnode"). Seems > nocache gives the best perf for me in this case. Sorry for jumping in here - I've got a couple of questions: - Will this also apply to nullfs read-only mounts? Or is it only in case of writing "through" a nullfs mount that these problems are seen? - Is it a problem also in 13, or is this "new" in -CURRENT? We're having weird and unexplained CPU spikes on several systems, even after tuning geli to not use gazillions of threads. So far our suspicion has been ZFS snapshot cleanups but this is an interesting contender - unless the whole "read only" part makes it moot. /Eirik