From owner-freebsd-fs@FreeBSD.ORG Wed Mar 19 12:51:08 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D1C94E5A for ; Wed, 19 Mar 2014 12:51:08 +0000 (UTC) Received: from r2-d2.netlabs.org (r2-d2.netlabs.org [213.238.45.90]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 42987122 for ; Wed, 19 Mar 2014 12:51:07 +0000 (UTC) Received: (qmail 41396 invoked by uid 89); 19 Mar 2014 12:51:05 -0000 Received: from unknown (HELO eternal.bfh.ch) (ml-ktk@netlabs.org@147.87.42.166) by 0 with ESMTPA; 19 Mar 2014 12:51:05 -0000 Message-ID: <532992B8.4090407@netlabs.org> Date: Wed, 19 Mar 2014 13:51:04 +0100 From: Adrian Gschwend User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem and fix References: <201403181520.s2IFK1M3069036@freefall.freebsd.org> <53288024.2060005@denninger.net> <53288629.60309@FreeBSD.org> In-Reply-To: <53288629.60309@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Mar 2014 12:51:08 -0000 On 18.03.14 18:45, Andriy Gapon wrote: >> This is consistent with what I and others have observed on both 9.2 >> and 10.0; the ARC will expand until it hits the maximum configured >> even at the expense of forcing pages onto the swap. In this >> specific machine's case left to defaults it will grab nearly all >> physical memory (over 20GB of 24) and wire it down. > Well, this does not match my experience from before 10.x times. I reported the issue on which Karl gave feedback and developed the patch. The original thread of my report started here: http://lists.freebsd.org/pipermail/freebsd-fs/2014-March/019043.html Note that I don't have big memory eaters like VMs, it's just a bunch of jails and services running in them. Including some JVMs. Check out the munin graphs before and after: Daily which does not seem to grow much anymore now: http://ktk.netlabs.org/misc/munin-mem-zfs1.png Weekly: http://ktk.netlabs.org/misc/munin-mem-zfs2.png You can actually see where I activated the patch (16.3), the system behaves *much* better since then. I did one more reboot that's why it goes down again but since then I did not reboot anymore. The moments where munin did not report anything the system was in the ARC-swap lock and virtually dead. From working on the system it feels like a new machine, everything is super fast and snappy. I don't understand much of the discussions you guys are having but I'm pretty sure Karl fixed an issue which gave me headache on BSD over years. I first saw this in 8.x when I started to use ZFS productively and I've seen it in all 9.x release as well up to this patch. regards Adrian