From nobody Fri Dec 8 10:17:49 2023 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmnC12kFdz53kq6 for ; Fri, 8 Dec 2023 10:17:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmnC06xHlz3dLX for ; Fri, 8 Dec 2023 10:17:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1702030669; a=rsa-sha256; cv=none; b=gcTP2Vfy6b1s1LqFYCdwQBMEUxKcGyS4WEosZMF5o0hzG9AGDXoYNM+PplZ19IwoFk3R/z 2jlt1BE5GSpCq9Jjn1y5IiZbF8EUCWlFEWLyZqzEnBwCShWxCKypqExz8O6hebZCTMzYHm MJJ76XcWsVLR5c5O823sEJG/ZUlrxx0dcsKn0QjE6SSn61Q/ZeqVZ6ox7C4KoD8c7PF1jn WfUU7EXrI3nYan/WXirONj4izUil8W6MQ0fusbZO0C/15ZZ2qSBQVDOjAhXtoJDR+ICkCv w8ZCbgzAsXIWBKn19ZmOtrfFBsLcAEIF/t2XulcuMJI1jQ1qmZH+itGqscvJvQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1702030669; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=isQALFR8YuRgEHeS4hOs5tGBJDKuKESqnGOFyipLsYQ=; b=TZMlS3gjWx3rAHc/oMrrk1sc400ncDF5VZcYGw9+/FO2H9NAtv1TWxMYEVVDfsE+fAdyGc wfHhl7VXjsX1S7HHTzJHLyalxz/9qrYJBRNaIEduHtLPavTD5PJUcyv24VV5Aoyi1apFa7 xPfn+Ok+7/2Ecadz45DXEC4CJ26tNOCyOPXc0fNzmHR5gY2TroroM1ezyvYak0cnUNHTYt N3LXX27MlWm2SZ4GpiXELszfC94NFkphE2p5V3WNIpBuGaU2XLMD7ZuRkc7lDqPN+tPIib JNe5yvdkqbZhMn0D1Icg2J+d2svikd2PSI5GaQE5dPjWmKnqngOAQrEZuElYww== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SmnC061Jdz1qw for ; Fri, 8 Dec 2023 10:17:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3B8AHmBf090019 for ; Fri, 8 Dec 2023 10:17:48 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3B8AHmax090018 for fs@FreeBSD.org; Fri, 8 Dec 2023 10:17:48 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 275594] High CPU usage by arc_prune; analysis and fix Date: Fri, 08 Dec 2023 10:17:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: seigo.tanimura@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D275594 --- Comment #6 from Seigo Tanimura --- (In reply to Seigo Tanimura from comment #5) The build under the following setting have completed: - vfs.vnode.vnlru.max_free_per_call: 10000 (out-of-box) - vfs.zfs.arc.prune_interval: 1000 (my fix enabled) Build time: 07:11:02 (292 pkgs / hr) Max vfs.vnode.stats.count: ~2.2M Max ARC memory size: ~5.6GB NB devel/ocl-icd failed because pkg-static was killed by the kernel for tak= ing too long to page in. 31 ports were skipped because of this failure. This error was often seen on 14.0-RELEASE-p0, indicating an obstacle upon the executable file access. This result is better than the baseline (14.0-RELEASE-p2) and worse than my original fix shown in the description. Although prune_interval avoided the contention upon vnode_list_mtx somehow, this setup also limited the ARC pru= ning performance, introducing another pressure including the overcommit upon the= ARC memory size. I conclude this setup is not optimal nor recommended. ----- Ongoing test: - vfs.vnode.vnlru.max_free_per_call: 4000000 (=3D=3D vfs.vnode.vnlru.max_free_per_call) - vfs.zfs.arc.prune_interval: 1000 (my fix enabled) This setup allows the unlimited workload to the ARC pruning under the configured interval. Another object of this test is the measurement of the vnode number ZFS requ= ests the OS to reclaim. As long as this value is below 100000 (vfs.vnode.vnlru.max_free_per_call in my first test), the system behaviour = and test results are expected to be the same as my first test. A glance on 30 minutes after the build start: - The activity of arc_prune is mostly the same as the first test; the CPU u= sage occasionally surges up to 30%, but it does not stay for more than 1 second = so far. - The average number of the vnodes ZFS requests to reclaim: ~44K. - vfs.vnode.stats.count: ~1.2M. - The default vfs.vnode.vnlru.max_free_per_call of 10K did regulate the A= RC pruning work. - I will keep my eyes on this figure, especially if it exceeds 100K. - The ARC memory size is strictly regulated as configured by vfs.zfs.arc_ma= x. - The ARC pruning starts when the ARC memory size reaches ~4.1GB. - The ARC pruning does not happen as long as the ARC memory size is below 4.0GB. The finding regarding the ARC memory size is something new to me. Maybe the vnode number requested for the reclaim by ZFS is calculated very carefully = and precisely, so we should actually honour that figure to keep the system heal= thy. I first treated this test as an extreme case, but maybe this should be evaluated as a working setup. --=20 You are receiving this mail because: You are the assignee for the bug.=