From owner-freebsd-bugs@freebsd.org Mon Jul 16 19:20:11 2018 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4595F1038A89 for ; Mon, 16 Jul 2018 19:20:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D24F47F993 for ; Mon, 16 Jul 2018 19:20:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 936A21038A85; Mon, 16 Jul 2018 19:20:10 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6ECEC1038A84 for ; Mon, 16 Jul 2018 19:20:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0564D7F98E for ; Mon, 16 Jul 2018 19:20:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 317DD15883 for ; Mon, 16 Jul 2018 19:20:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w6GJK9fl088338 for ; Mon, 16 Jul 2018 19:20:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w6GJK9h0088336 for bugs@FreeBSD.org; Mon, 16 Jul 2018 19:20:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 229670] ZFS ARC limit vfs.zfs.arc_max from /boot/loader.conf is not respected Date: Mon, 16 Jul 2018 19:20:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: leif@ofWilsonCreek.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2018 19:20:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229670 --- Comment #7 from Leif Pedersen --- The machines I have observed this on vary in zpool sizes. With regard to the "rule of thumb", one machine which behaves particularly horribly has a single zpool sized at 256GB. It has only 10GB referenced (meaning non-snapshot data) and less than 20k inodes. A linear interpretati= on of the rule of thumb suggests that just 10MB should be enough ARC, although= I don't expect it to scale down that low. On this one, arc_max is set to 256M= B, but the ARC runs well over 1 GB. I don't know how high it would go if left alone, since it only has 2 GB of RAM to begin with, so when it gets that bi= g I have to reboot it. This one is an AWS VM. For another example, I have a physical machine with 6GB of RAM, with arc_max set to 256MB and top showing the ARC at 2GB. This one is a bit bigger -- it= has 1.4TB across 2 zpools. It does rsync-style backups for three other machines= , so there's a relatively large number of filenames. The second zpool (for the backups) has roughly 5M inodes with roughly 70-75M filenames (up to 15 names per inode), with most of its inodes read in a short time span. However, I've been running this system with these backups on ZFS for years, at least as f= ar back as FreeBSD 9, without memory problems. While it isn't a huge system, it was always very stable in the past. While I don't see this issue on larger machines (with 128GB RAM or more, for example), I don't believe this is about a minimum memory requirement for a = few reasons. To begin with, the machines are not insanely tiny or running with a wildly unbalanced disk/ram ratio. Also, if there's a hard minimum requireme= nt, then sysctl should throw an error. Also, sysctl reports vfs.zfs.arc_meta_li= mit at ~67MB on both, which is much lower than arc_max. However, I retract my remark about it maybe being from a recent update, bec= ause uname on the AWS machine reports 11.1-RELEASE-p4. (I often don't reboot aft= er updating unless the kernel has a serious vulnerability, and this one has be= en up for 109 days.) Again, mine are 11.1 with the latest patches by freebsd-update. I could try upgrading to 11.2 if it would be an interesting data point. >The patch in review is about ARC releasing its cache... This patch would likely help, particularly since these examples don't have swap. It seems likely to alleviate my need to meddle with arc_max, which wo= uld be great. However, I'd argue that it's still a bug that arc_max is apparent= ly completely ignored. And now that I think about that, it's also still bug th= at OOM-killing processes is preferred to swap OR evacuating ARC, unless that p= atch is fixing that also. I'd swear I remember that fairly recently, I tried changing arc_max and top immediately showed the ARC chopped off at the new setting, and if I remember that right then this is clearly a regression...but details of that memory a= re vague at this point. --=20 You are receiving this mail because: You are the assignee for the bug.=