From nobody Fri Oct 27 15:07:13 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SH5cZ0S6Xz4yLjX for ; Fri, 27 Oct 2023 15:07:26 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SH5cX2bz3z4fJx for ; Fri, 27 Oct 2023 15:07:24 +0000 (UTC) (envelope-from mad@madpilot.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=madpilot.net header.s=bjowvop61wgh header.b="Q TmLKV8"; spf=pass (mx1.freebsd.org: domain of mad@madpilot.net designates 159.69.1.99 as permitted sender) smtp.mailfrom=mad@madpilot.net; dmarc=pass (policy=quarantine) header.from=madpilot.net Received: from mail (mail [IPv6:fd5c:5351:d272::3]) by mail.madpilot.net (Postfix) with ESMTP id 4SH5cP1S3xz6ddR for ; Fri, 27 Oct 2023 17:07:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=madpilot.net; h= content-transfer-encoding:content-type:content-type:in-reply-to :from:from:references:content-language:subject:subject:date:date :message-id:received; s=bjowvop61wgh; t=1698419234; x= 1700233635; bh=WfVUpkQIh0b18urEHWVkBPCaSQ2bA04Ln2a8YuX5iec=; b=Q TmLKV8ZwHgWTmROdimwFUDL0tjHt83eICDNfGaG1rmo6n012Jk4PXFnotxFWxYZB x58v1BpTt0kem3LvSxiu3a3NGMPeZ5aXCaxFsSsrSYZsKlmpKvXt22sGV5In8X20 jmHR/UAhe92CwRN4cJzZ56LeB4T4uOIdjTK6OB5r5MuF5hGFLHO2RsjcDVaBp9sc I/QE2RjRfiRo89os8Ult5Kp4ds+IqUxfv+4/QjwWMim1km5Zzz0JvFmnROfA2U6y jfe4y2CQeTPh83Q5qHUf/kJwIznAalZo1rlTWlSidx9rvnXVdI59M57JunSyXOdm 4piyLxJGl0fjITBXTxLhw== Received: from mail.madpilot.net ([IPv6:fd5c:5351:d272::3]) by mail (mail.madpilot.net [IPv6:fd5c:5351:d272::3]) (amavisd-new, port 10026) with ESMTP id 9uGvCFsML7XW for ; Fri, 27 Oct 2023 17:07:14 +0200 (CEST) Message-ID: Date: Fri, 27 Oct 2023 17:07:13 +0200 Subject: Re: periodic daily takes a very long time to run (14-stable) Content-Language: en-US, it To: freebsd-stable@freebsd.org References: <1122335317.4913.1698407124469@localhost> <794932758.6659.1698413675475@localhost> From: Guido Falsi Autocrypt: addr=mad@madpilot.net; keydata= xsBNBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAHNHkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PsLAeAQTAQIAIgUCT4b6XQIbAwYL CQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQGuaGDlbL0pOWigf/YVTVf3+ZRnzeGP7CjGV1 Wrrxzjc8h8W64NZasV0XLHGFjl5MYwtm9jJ9gbL8Ubtqstey7lYpjOk2fG6YDhY5eptWCpR6 1QqYrioukhCfKbodSk6PnIZcx719nJVK2P7ihdFEN78TavpBwqIf9hGEcKkMpbRFQv1mYvXD hKVwQGY+8bkH/a/pAWmIyD4qMfKCMurH5DexxEt5SYWu5BB5hd/DWyZ0wuZ+F79KMPzLBPJW 5cpdLNbrvenSqFZGJEGhtTp7GFJJr6lTy8VLBArxmFHiY5jGyR45eZEGDcz86FfGgvPnnpi7 aNCc/ROdF7fnZYPh8uZGGjQbd4EYK4xMzc7BTQRTEHtBARAAoWGsNx6g90r8gcNKaiPpJBiK y8ztV2FyV5LsT0OgQBW3vIxt/odtsxVNNjpyS/BNZCyzLAsFc1WrGBzhYsmPN9SGB5/5YTvk zf5YViU5VAsZlj/MRWCZrWtpic4c0A7N4csOYReNtk/q8YB4PIFsZ9A+kTuoZhnu5t5PdfBA 74+SVwKu84+PZk9wDEY1LbFVT8vM42oKsmoswlIhwJ2xuJI/gbk+cMUe0yiRpNjo4Svw4RB8 4B6uFwdRr/PtS7xi2Zqoof5AaQT9YSBpGpKJOe/Qk5MP4PF6Fqq+go89n77Y2kJkwcHaLoD/ GJ+ZDASIiMRe1y54FHOQ1RCTGGpnJLXdKuGhwv3J21pU8HNlq0ASNQMMQmYAwtUWzjmp/KEy I1qkcmjafcxb8TmiaoK8SQN1Zf96fc/sIrZN6Z5oOCEyyCQ0prH/PTA2jlRkKQ487PTGk2JS KU5VuS57Nlk2DrnvjWp57aV9eFAhpnrrJPuGmFz83/Pc8gC0t7N7i7VVHYRcC5naxYB2UoI1 OUkyxpT/HvQFXXVZ3/KmdXMzrx191AggCPWIwUAP+VcaURSYpeDk6/ZVAOVOe1ChqcJisCD7 wK20/OOvJ2AtkWreGu1CZ9zSx7nK/VYdLr34GxQ4bT1G+9rBQNnFSNbX2TJ431Mdo1GCjDeR K4CtSnrNKYkAEQEAAcLAXwQYAQgACQUCUxB7QQIbDAAKCRAa5oYOVsvSkw3nCADhsKRf+rAR ULTpOh5HoLam62ZJZAyCkNqqu/rke5uj5AaaDY/h7BNhBDiDqhhZLTeofGpVVaErPsWN+tX5 0fypsIt9KAhy90GFrtrIZlWuyK4wsoZvDfp9yaRk+lIM58dw/Rcfxn670JaPTFSRPECVn/uL qBhJSkbYlY212YT9fxVUTJe6wIvDLQrQEjrQD/h1FMhfcLhAqsndltRd6DPvTKeMd/6VAxn0 hkoBKhEy5LkWjM9CHppu+bBkQ91/kj2uJQSXO8euonwHHS3c+6N2i2H7I0emcHGu07wuRB2t Dnw/RLBxohffdPZT2kbxuG7lhVHzwVDw5DRwSw8GkOdy In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-2.00 / 15.00]; MISSING_MIME_VERSION(2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[madpilot.net,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[madpilot.net:s=bjowvop61wgh]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; BLOCKLISTDE_FAIL(0.00)[159.69.1.99:server fail]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[madpilot.net:+]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4SH5cX2bz3z4fJx X-Spamd-Bar: - List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org On 27/10/23 16:42, void wrote: > On Fri, Oct 27, 2023 at 03:34:35PM +0200, Ronald Klop wrote: >> So if my observation is right it might be interesting to find out what >> is writing. > > would ktrace and/or truss be useful? something else? The truss -p output > of the > find PID produces massive amounts of output, all like this: > > fstatat(AT_FDCWD,"5e70d5f895ccc92af6a7d5226f818b-81464.o",{ > mode=-rw-r--r-- ,inode=367004,size=10312,blksize=10752 > },AT_SYMLINK_NOFOLLOW) = 0 (0x0) > > with the filename changing each time > > (later...) > > that file is in ccache!!! > > locate 5e70d5f895ccc92af6a7d5226f818b-81464.o > /var/cache/ccache/f/5/5e70d5f895ccc92af6a7d5226f818b-81464.o > > maybe if I can exclude that dir (and /usr/obj) it'll lessen the periodic > runtime. > But i don't know yet whats calling find(1) when periodic daily runs. If > I can, I might be able to tell it not to walk certain heirarchies. I stumbled on a similar problem. Periodic trying to scan huge ccache directories. The script doing that was locate database indexing in weekly. Not sure if you're having the same issue, but this could give you ideas. My fix was: weekly_locate_enable="YES" PRUNEPATHS="/tmp, /usr/tmp, /var/tmp, /var/db/portsnap, /var/db/freebsd-update, /dumpster/ccache, /poudriere" There is no advantage in locate indexing ccache, and I don't want it to cache poudriere either. (mhmm, that portsnap directory looks like a leftover from the past...) Also, unluckily poudriere in modern times does a lot of disk IO (old plain ports building too BTW) and it is really best, especially for SSD media, to have enough ram to run the full build there. Will really save on SSD writes. -- Guido Falsi