From owner-freebsd-fs@FreeBSD.ORG Wed Aug 11 19:25:42 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D9601106566C for ; Wed, 11 Aug 2010 19:25:42 +0000 (UTC) (envelope-from marco@tolstoy.tols.org) Received: from tolstoy.tols.org (tolstoy.tols.org [IPv6:2a02:898:0:20::57:1]) by mx1.freebsd.org (Postfix) with ESMTP id 5B0A88FC08 for ; Wed, 11 Aug 2010 19:25:42 +0000 (UTC) Received: from tolstoy.tols.org (localhost [127.0.0.1]) by tolstoy.tols.org (8.14.4/8.14.4) with ESMTP id o7BJPcsf045085 for ; Wed, 11 Aug 2010 19:25:38 GMT (envelope-from marco@tolstoy.tols.org) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.96.1 at tolstoy.tols.org Received: (from marco@localhost) by tolstoy.tols.org (8.14.4/8.14.4/Submit) id o7BJPbQQ045084 for freebsd-fs@freebsd.org; Wed, 11 Aug 2010 19:25:37 GMT (envelope-from marco) Date: Wed, 11 Aug 2010 19:25:37 +0000 From: Marco van Tol To: freebsd-fs@freebsd.org Message-ID: <20100811192537.GA44635@tolstoy.tols.org> Mail-Followup-To: freebsd-fs@freebsd.org References: <20100810214418.GA28288@tolstoy.tols.org> <20100811014919.GA52992@icarus.home.lan> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100811014919.GA52992@icarus.home.lan> User-Agent: Mutt/1.4.2.3i X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on tolstoy.tols.org Subject: Re: zfs arc - just take it all and be good to me X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2010 19:25:43 -0000 On Tue, Aug 10, 2010 at 06:49:19PM -0700, Jeremy Chadwick wrote: > On Tue, Aug 10, 2010 at 09:44:18PM +0000, Marco van Tol wrote: > > [...] > > > > All in all this looks like a close attempt at zfs memory being auto > > tuned while using maximum amount of memory. The only problem is, nobody > > else is doing it like this so its very likely that this is not the smart > > thing to do. > > I'm not sure what "nobody else is doing it like this" means, but Solaris > 10 behaves exactly as you describe -- the ARC takes up as much memory as > it can. When an application or other piece of the kernel wants memory > which the ARC can free up, it releases memory. There's no tuning > required on Solaris; it "just works". Hm by "nobody else is doing it like this" I meant on FreeBSD. I also meant to say that what you will typically see people advising/doing is setting arc_max to a value much less then their total physical memory. What I have done instead is set arc_max to nearly all my physical memory (1792G out of a total of 2GB). In combination with that I set kmem_size to 1.5x physical memory to account for kernel fragmentation. Here is a post that touches that subject: http://lists.freebsd.org/pipermail/freebsd-fs/2010-January/007614.html What I am glad to see happening so far, but I haven't given the system any production levels of stress, is that the arc happely takes up nearly all my memory, and just surrenders it to programs that happen to run, taking it back after they finish. Very nice. (I'll feed it bonny++ and see what happens) My mail wasn't intended to make any complaints, much the contrairy. I am a happy camper ever since I changed the test system to those settings. I think/hope that was clear. :) > At my day job (where we use Solaris 10), when we introduced ZFS into the > picture, many of our memory usage monitors began firing indicating lack > of free memory due to ARC usage. We had to add some code to our > check_system_memory monitor which called kstat to examine the > zfs:0:arcstats:size and zfs:0:arcstats:c_min properties and then add > that to the overall amount of memory available. Code in question: > > my $zfsarcstatssize = "zfs:0:arcstats:size"; > my $zfsarcstatsmin = "zfs:0:arcstats:c_min"; > my $zfsarcsize; > my $zfsarcmin = 0; > if ($OS eq 'solaris' and -x $kstat) { > my @dump = `$kstat -p $zfsarcstatssize $zfsarcstatsmin`; > unless ($?) { > foreach my $dump (@dump) { > chomp $dump; > my ($label, $size) = split(/\t/, $dump); > if ($label eq $zfsarcstatssize) { > $zfsarcsize = sprintf("%.2f", $size/1024/1024); > } > if ($label eq $zfsarcstatsmin) { > $zfsarcmin = sprintf("%.2f", $size/1024/1024); > } > } > $real_available += $zfsarcsize - $zfsarcmin if (defined $zfsarcsize and $zfsarcsize > $zfsarcmin); > } > } Its nice when you can rely on something to act equally to actual free memory, yet provide great speedups at times it can use that memory. > I believe OpenSolaris behaves the same as Solaris 10 in this regard. > Their VM model differs from that of FreeBSD. > > I've talked about this in the past (see paragraph starting with "Fast > forward to today"). I can dig up John's original response (talking > about how the VM needs to be improved in FreeBSD to accomplish the > equivalent of what Solaris does) if need be. > > http://lists.freebsd.org/pipermail/freebsd-fs/2010-May/008598.html I had seen that post before indeed. Thank you. The bottom line of my original mail was: - Hey, I set the arc to just take all my RAM - It grows on reads and writes - It shrinks when programs need the memory it has. - It behaves well in a scenario of combined read/write/other programs. - This is good, really good, what is dangerous about my settings that not everybody just sets the arc to nearly all their physical memory on FreeBSD? That's all, :) Thanks, Marco -- A male gynecologist is like an auto mechanic who never owned a car. - Carrie Snow