From nobody Fri Oct 27 14:42:00 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SH53K5g0Wz4yKJw for ; Fri, 27 Oct 2023 14:42:05 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SH53J6Jwfz4b0B for ; Fri, 27 Oct 2023 14:42:04 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b="mzFA/Yyd"; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="cUcJ5W7/"; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.28 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 780225C0166 for ; Fri, 27 Oct 2023 10:42:04 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Fri, 27 Oct 2023 10:42:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1698417724; x=1698504124; bh=7u dlC+ES+ZjTsKvnc84mcPQs/pi/mrMqAX6gYtFaEbI=; b=mzFA/YydVN87nguqhv Lwi2nBVEABMqoe68/gaxtXzt7kSX3Fp3IHlZl663bt9aV2AlhpGVNPUrJCWbh+Mu 92ODLJ40GCWUwhcphIQC5pPyHHGZRkye+8gmSQbrRVRKIL31AsFh6M2AzLRujAoV IOSUMntN1qKXZySIhdQMGiSxV7jFQZYCsUjZzuZIvBoXlMBDvbLWq2ASypFNK558 vMnV9HyVAG5ngm19Utbh6+w+Zd+Rt2ywlcr4JUJQPX5axgeMMw4hmoMofuLSlQmP TJk8Q6DN2FqVqPZrVpH3geTJiKR9J+Lq55wHdRlJ9830nOO6tQsp7wZUVjn0wYv5 9bQg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698417724; x=1698504124; bh=7udlC+ES+ZjTs Kvnc84mcPQs/pi/mrMqAX6gYtFaEbI=; b=cUcJ5W7/CeTn9KeKttTHe0QIkx6Lm 7KpJWPp3VFpYoZIvyl29zLBqTA2vyyTJvlkLhnKwscevUrrY642tl74UcbHqL1lD ClTUer8W/3M5ajc7JTf4iPS5N+oDgyNIfzMzOm/nI0/7hCRcG5LGEwhC0Ji85WpB hNw+bHWHUyIRlnt5QimIazIgS5xpgiwjHEX1WRM8qNA6A1gyKseCJv3T0MnZbRkx r6Vb03xk6RJmQoXccdDx8V7hmOjFb35GnzKyanzKF53u9uCT818MoFJdwnfb18Y7 SszZTVcHKYKa/CYXASzC7y6ncVtaoTml18ryosKnpyfLuL+KrGvNZfAUg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrleeggdektdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehttdertd dttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrght thgvrhhnpeejudeuiefffedtteehudehgeeukeetffdvfeevieefhefhgfffteduueehte eikeenucffohhmrghinhepfihikhhiphgvughirgdrohhrghenucevlhhushhtvghrufhi iigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 27 Oct 2023 10:42:03 -0400 (EDT) Date: Fri, 27 Oct 2023 15:42:00 +0100 From: void To: freebsd-stable@freebsd.org Subject: Re: periodic daily takes a very long time to run (14-stable) Message-ID: Mail-Followup-To: freebsd-stable@freebsd.org References: <1122335317.4913.1698407124469@localhost> <794932758.6659.1698413675475@localhost> List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <794932758.6659.1698413675475@localhost> X-Spamd-Result: default: False [-4.69 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.987]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.28:from]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SH53J6Jwfz4b0B X-Spamd-Bar: ---- On Fri, Oct 27, 2023 at 03:34:35PM +0200, Ronald Klop wrote: > >What stands out to me is that you do quite a lot of writes on the disk. (I might be mistaken.) >The max. number of IOPS for HDD is around 80 for consumer grade harddisks. I think this counts for USB connected disks. >https://en.wikipedia.org/wiki/IOPS#Mechanical_hard_drives >>From the stats you posted it looks like you are almost always doing 50+ writes/second already. That does not leave much IOPS for the find process. > >ZFS tries to bundle the writes every 5 seconds to leave room for the reads. See "sysctl vfs.zfs.txg.timeout". Unless it has too much data to write or a sync request comes in. % sysctl vfs.zfs.txg.timeout vfs.zfs.txg.timeout: 5 do I need to tune this? Here's equivalent output from my setup (I ran periodic daily again) #device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b da0 16 18 191.9 557.9 50 8 144 29 10 24 da0 107 0 699.7 0.0 52 0 0 52 1 99 da0 102 0 409.2 0.0 71 0 0 71 2 98 da0 65 6 259.6 49.4 101 143 0 105 12 101 da0 57 14 227.7 123.9 153 163 0 155 12 100 da0 40 19 158.8 285.8 205 103 0 172 12 98 da0 46 30 191.1 441.9 180 58 0 132 11 91 da0 63 4 261.6 16.1 162 250 239 170 6 112 da0 67 10 273.7 83.6 99 66 0 95 12 91 da0 32 21 129.4 177.9 223 102 0 175 5 97 da0 48 16 191.9 261.3 173 130 0 162 9 109 da0 38 19 152.2 191.3 168 61 292 139 8 104 da0 92 0 366.9 0.0 104 0 0 104 4 100 da0 73 10 291.7 87.9 76 99 0 79 12 97 da0 49 15 195.2 270.9 156 129 0 150 11 103 da0 53 15 212.3 248.3 139 128 0 137 12 92 da0 54 22 216.1 272.1 151 81 92 130 8 107 da0 80 4 320.9 16.0 74 201 125 80 3 100 da0 55 10 218.8 72.9 89 73 0 87 11 82 ^C % zpool iostat 1 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- zroot 93.6G 818G 13 16 161K 506K zroot 93.6G 818G 91 0 367K 0 zroot 93.6G 818G 113 0 454K 0 zroot 93.6G 818G 102 0 411K 0 zroot 93.6G 818G 98 0 422K 0 zroot 93.6G 818G 67 18 271K 171K zroot 93.6G 818G 43 16 173K 252K zroot 93.6G 818G 43 28 173K 376K zroot 93.6G 818G 78 3 315K 15.9K zroot 93.6G 818G 94 0 378K 0 zroot 93.6G 818G 103 0 414K 0 zroot 93.6G 818G 102 0 658K 0 zroot 93.6G 818G 98 0 396K 0 zroot 93.6G 818G 109 0 438K 0 zroot 93.6G 818G 101 0 404K 0 zroot 93.6G 818G 47 13 191K 91.4K zroot 93.6G 818G 52 11 209K 126K zroot 93.6G 818G 50 20 202K 301K zroot 93.6G 818G 46 12 186K 128K zroot 93.6G 818G 86 0 346K 3.93K zroot 93.6G 818G 45 18 183K 172K zroot 93.6G 818G 42 15 172K 343K zroot 93.6G 818G 43 24 173K 211K zroot 93.6G 818G 87 0 596K 0 ^C >So if my observation is right it might be interesting to find out what is writing. would ktrace and/or truss be useful? something else? The truss -p output of the find PID produces massive amounts of output, all like this: fstatat(AT_FDCWD,"5e70d5f895ccc92af6a7d5226f818b-81464.o",{ mode=-rw-r--r-- ,inode=367004,size=10312,blksize=10752 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) with the filename changing each time (later...) that file is in ccache!!! locate 5e70d5f895ccc92af6a7d5226f818b-81464.o /var/cache/ccache/f/5/5e70d5f895ccc92af6a7d5226f818b-81464.o maybe if I can exclude that dir (and /usr/obj) it'll lessen the periodic runtime. But i don't know yet whats calling find(1) when periodic daily runs. If I can, I might be able to tell it not to walk certain heirarchies. >I had similar issues after the number of jails on my RPI4 increased and they all >were doing a little bit of writing which accumulated in quite a lot of writing. I'm at a loss as to what's doing the writing. The system runs the following: poudriere-devel # for aarch64 and armv7 apcupsd # for ups monitoring vnstat # bandwidth use, writes to its db in /var/db/vnstat sshd exim (local) pflogd # right now it's behind a firewall, on NAT so it's not doing much pf # same ntpd powerd nginx # this serves the poudriere web frontend, and that's it (http-only) syslogd >My solution was to add an SSD. I have an nfs alternative. The LAN is 1GB. But I think the fix will be to tell find to not search some paths. just need to work out how to do it. What would the effect be of increasing or decreasing the txg delta with system performance? -- >