From owner-freebsd-bugs@freebsd.org Tue Oct 13 19:00:59 2020 Return-Path: Delivered-To: freebsd-bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 22B0E44895F for ; Tue, 13 Oct 2020 19:00:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4C9lHv07Kcz4756 for ; Tue, 13 Oct 2020 19:00:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 02850448E06; Tue, 13 Oct 2020 19:00:59 +0000 (UTC) Delivered-To: bugs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 024BE448E05 for ; Tue, 13 Oct 2020 19:00:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C9lHt6Mtwz47K7 for ; Tue, 13 Oct 2020 19:00:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BEA6A2220B for ; Tue, 13 Oct 2020 19:00:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 09DJ0wZi063045 for ; Tue, 13 Oct 2020 19:00:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 09DJ0wFY063025 for bugs@FreeBSD.org; Tue, 13 Oct 2020 19:00:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 250323] OpenZFS L2ARC size shrinking over time Date: Tue, 13 Oct 2020 19:00:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: se@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2020 19:00:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D250323 Bug ID: 250323 Summary: OpenZFS L2ARC size shrinking over time Product: Base System Version: CURRENT Hardware: Any OS: Any Status: New Severity: Affects Many People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: se@FreeBSD.org After the switch-over to OpenZFS in -CURRENT I have observed that the L2ARC shrinks over time (at a rate of 10 to 20 MB/day). My system uses a 1 TB NVME SSD partitioned as 64 GB of SWAP (generally unused) and 256 GB of ZFS cache (L2ARC) to speed up reads from a 3*6 TB raidz1. (L2ARC persistence is great, especially on a system that is used for development and rebooted into the latest -CURRENT about once per week!) After reboot, the full cache partition is available, but even measured only minutes apart the reported site of the L2ARC is declining. The following two values were obtained just 120 seconds apart: kstat.zfs.misc.arcstats.l2_asize: 273831726080 kstat.zfs.misc.arcstats.l2_asize: 273831644160 [After finishing the text of this mail I have checked the value of that variable another time - maybe 10 minutes have passed ... kstat.zfs.misc.arcstats.l2_asize: 273827724288 That corresponds with some 4 MB lost over maybe 10 minutes ...] I have first noticed this effect with the zfs-stats command updated to support the OpenZFS sysctl variables (committed to ports a few days ago). After 6 days of uptime the output of "uptime; zfs-stats -L" is: 12:31PM up 6 days, 7 mins, 2 users, load averages: 2.67, 0.73, 0.36 ------------------------------------------------------------------------ ZFS Subsystem Report Mon Oct 12 12:31:57 2020 ------------------------------------------------------------------------ L2 ARC Summary: (HEALTHY) Low Memory Aborts: 87 Free on Write: 5.81 k R/W Clashes: 0 Bad Checksums: 0 IO Errors: 0 L2 ARC Size: (Adaptive) 160.09 GiB Decompressed Data Size: 373.03 GiB Compression Factor: 2.33 Header Size: 0.12 458.14 MiB L2 ARC Evicts: Lock Retries: 61 Upon Reading: 9 L2 ARC Breakdown: 12.66 m Hit Ratio: 75.69% 9.58 m Miss Ratio: 24.31% 3.08 m Feeds: 495.76 k L2 ARC Writes: Writes Sent: 100.00% 48.94 k ------------------------------------------------------------------------ After a reboot and with the persistent L2ARC now reported to be available again (and filled with the expected amount of data): 13:24 up 28 mins, 2 users, load averages: 0,09 0,05 0,01 ------------------------------------------------------------------------ ZFS Subsystem Report Mon Oct 12 13:24:56 2020 ------------------------------------------------------------------------ L2 ARC Summary: (HEALTHY) Low Memory Aborts: 0 Free on Write: 0 R/W Clashes: 0 Bad Checksums: 0 IO Errors: 0 L2 ARC Size: (Adaptive) 255.03 GiB Decompressed Data Size: 633.21 GiB Compression Factor: 2.48 Header Size: 0.14 901.41 MiB L2 ARC Breakdown: 9.11 k Hit Ratio: 35.44% 3.23 k Miss Ratio: 64.56% 5.88 k Feeds: 1.57 k L2 ARC Writes: Writes Sent: 100.00% 205 ------------------------------------------------------------------------ After 32 hours of uptime with only light load: 255 GB -> 242 GB 8:58PM up 1 day, 8:01, 1 user, load averages: 4.03, 1.02, 0.41 ------------------------------------------------------------------------ ZFS Subsystem Report Tue Oct 13 20:58:02 2020 ------------------------------------------------------------------------ L2 ARC Summary: (HEALTHY) Low Memory Aborts: 7 Free on Write: 78 R/W Clashes: 0 Bad Checksums: 0 IO Errors: 0 L2 ARC Size: (Adaptive) 242.33 GiB Decompressed Data Size: 603.73 GiB Compression Factor: 2.49 Header Size: 0.13% 828.25 MiB L2 ARC Evicts: Lock Retries: 4 Upon Reading: 0 L2 ARC Breakdown: 1.34 m Hit Ratio: 74.50% 1.00 m Miss Ratio: 25.50% 342.50 k Feeds: 110.24 k L2 ARC Writes: Writes Sent: 100.00% 11.82 k ------------------------------------------------------------------------ I do not know whether this is just an accounting effect, or whether the usable size of the L2ARC is actually shrinking, but since there is data in the L2ARC after the reboot, I assume it is just an accounting error. But I think this should still be researched and fixed - there might be a wrap-around after several weeks of up-time, and if the size value is not only used for display purposes, this might lead to unexpected behavior. --=20 You are receiving this mail because: You are the assignee for the bug.=