From owner-freebsd-fs@FreeBSD.ORG Fri Mar 2 11:11:33 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD55A106566B for ; Fri, 2 Mar 2012 11:11:33 +0000 (UTC) (envelope-from peterjeremy@acm.org) Received: from mail14.syd.optusnet.com.au (mail14.syd.optusnet.com.au [211.29.132.195]) by mx1.freebsd.org (Postfix) with ESMTP id 514BE8FC12 for ; Fri, 2 Mar 2012 11:11:32 +0000 (UTC) Received: from server.vk2pj.dyndns.org (c220-239-116-103.belrs4.nsw.optusnet.com.au [220.239.116.103]) by mail14.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id q22BBSl0008647 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 2 Mar 2012 22:11:30 +1100 X-Bogosity: Ham, spamicity=0.000000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.5/8.14.4) with ESMTP id q22BBQwJ042605; Fri, 2 Mar 2012 22:11:26 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.5/8.14.4/Submit) id q22BBOOh042604; Fri, 2 Mar 2012 22:11:24 +1100 (EST) (envelope-from peter) Date: Fri, 2 Mar 2012 22:11:23 +1100 From: Peter Jeremy To: Luke Marsden Message-ID: <20120302111123.GA40708@server.vk2pj.dyndns.org> References: <1330081612.13430.39.camel@pow> <20120227181436.GA49667@server.vk2pj.dyndns.org> <20120301142826.GG97848@zxy.spb.ru> <20120302102509.Horde.6uPSdpjmRSRPUJH1lHEHc3A@webmail.leidinger.net> <1330683366.3819.20.camel@pow> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="17pEHd4RhPHOinZp" Content-Disposition: inline In-Reply-To: <1330683366.3819.20.camel@pow> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, Alexander Leidinger Subject: Re: Another ZFS ARC memory question X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Mar 2012 11:11:34 -0000 --17pEHd4RhPHOinZp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable [Cc list pruned] On 2012-Mar-02 10:16:06 +0000, Luke Marsden = wrote: >We are presently working around it by limiting arc_max to 4G on our 24G >RAM production boxes (which seems like a massive waste of performance) >and by doing very careful/aggressive application level management of >memory usage to ensure stability (limits.conf didn't work for us, so we >rolled our own). A better solution would be welcome, though, so that we >can utilise all the free memory we're presently keeping around as a >safety margin - ideally it would be used as ARC cache. Have you tried increasing vm.v_cache_min to cover your spikes? >1. Is it expected that with a 4G limited arc_max value that we should >see wired memory usage around 7-8G? I understand that the kernel has to >use some memory, but really 3-4G of non-ARC data? Yes, that sounds possible. >2. We have some development machines with only 3G of RAM. Previously >they had no arc_max set and were left to tune themselves. They were >quite unstable. Now we've set arc_max to 256M but things have got >worse: we've seen a disk i/o big performance hit (untarring a ports >tarball now takes 20 minutes), and wired memory usage is up around >2.5GB, the machines are swapping a lot, and crashing more frequently. That's stress-testing ZFS more than anything else. You definitely can't use those results as a guide to tune your production boxes (other than what not to do). That said, I have 3.5GB in my $work desktop (running 8.2-stable from about a month ago) and don't have any stability issues with either it or a buildbox with 2GB RAM. >Follows is arc_summary.pl from one of the troubled dev machines showing >the ARC using 500% of the memory it should be. Also uname follows. My >second question is: have there been fixes between 8.2-RELEASE and >8.3-BETA1 or 9.0-RELEASE which solve this ARC over-usage problem? There definitely have been some commits to ensure that arc_max is treated much more as a hard limit but I can't quickly find them so I'm not sure if they pre- or post-date 8.2-RELEASE. --=20 Peter Jeremy --17pEHd4RhPHOinZp Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk9QqtsACgkQ/opHv/APuIfWmgCfW+tqI1+n5qV9b5dWZ4F2aK/M khEAoKqIgtv3M7cmbCQ0RBRrIJpSEfwj =DgVI -----END PGP SIGNATURE----- --17pEHd4RhPHOinZp--