From owner-freebsd-fs@FreeBSD.ORG Sat Mar 5 15:10:29 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECB311065673 for ; Sat, 5 Mar 2011 15:10:29 +0000 (UTC) (envelope-from andriy@irbisnet.com) Received: from smtp104.rog.mail.re2.yahoo.com (smtp104.rog.mail.re2.yahoo.com [206.190.36.82]) by mx1.freebsd.org (Postfix) with SMTP id 965368FC13 for ; Sat, 5 Mar 2011 15:10:29 +0000 (UTC) Received: (qmail 52096 invoked from network); 5 Mar 2011 14:43:49 -0000 Received: from smtp.irbisnet.com (andriy@99.235.226.221 with login) by smtp104.rog.mail.re2.yahoo.com with SMTP; 05 Mar 2011 06:43:46 -0800 PST X-Yahoo-SMTP: dz9sigaswBA5kWoYWVTZrGHmIs2vaKgG1w-- X-YMail-OSG: 7DBWu_IVM1kvxl2EtZHWY8pdoAXvVuPAJ.MgF8KGhlnCtgs Kw_A6yFG9141ivX32C1dS9tihZqInnIf2oAoeTjsLA8QqdgDjekSxofMjXNt mB9JGxlkTWruhu.GCKXVInVExNjvRwlKJr7Q.naQbIxAcuDrXk8ikpFOzdqU 1bCtAmsSI6eLIEMhx4ZdwDbHW9DShQvJ997hwFc4lFUCTX.Q4JxFIs5lp9O1 FX_1Ed0X7gPEg.hiAVc8JbhI82AUh5X7nx6urZyl.Mryx X-Yahoo-Newman-Property: ymail-3 Received: from [192.168.0.40] (unknown [192.168.0.40]) by smtp.irbisnet.com (Postfix) with ESMTPSA id 063F22658F; Sat, 5 Mar 2011 09:43:45 -0500 (EST) References: <1299232133.18671.3.camel@pc286.embl.fr> <4D70E7D1.5030205@egr.msu.edu> <20110304143801.GA1734@icarus.home.lan> In-Reply-To: <20110304143801.GA1734@icarus.home.lan> Mime-Version: 1.0 (iPhone Mail 8C148) Message-Id: <625F7D8A-78F5-426B-B2C3-A1B4D289FD5F@irbisnet.com> X-Mailer: iPhone Mail (8C148) From: Andriy Bakay Date: Sat, 5 Mar 2011 09:42:47 -0500 To: Jeremy Chadwick Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-fs@freebsd.org" , "pjd@FreeBSD.org" Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Mar 2011 15:10:30 -0000 On 2011-03-04, at 9:38, Jeremy Chadwick wrote: > On Fri, Mar 04, 2011 at 08:23:29AM -0500, Adam McDougall wrote: >> On 03/04/11 04:48, Micka=C3=ABl Can=C3=A9vet wrote: >>>> I'd use vm.kmem_size=3D"32G" (i.e. twice your RAM) and that's it. >>>=20 >>> Should I also increase vfs.zfs.arc_max ? >>>=20 >>> Do you have any idea why the kernel panicked at only 8GB allocated ? >>>=20 >>> Thank you >>=20 >> I believe ARC allocations in kmem can become fragmented, so when it >> is searching for a place to store a new contiguous segment of >> memory, the remaining fragmented free spaces may all be too small. >> I also set >> vm.kmem_size to about twice the amount of ram to help it avoid this >> issue. I suspect if kmem is badly fragmented then performance of >> ZFS can downgrade, so that is another reason to keep kmem bigger. >=20 > My findings on 8.2-RELEASE indicate that doing this results in very > unexpected behaviour regarding the ARC maximum. As such, I cannot > recommend this model. >=20 > For example, on an amd64 system with 8GB physical RAM and these two > settings in /boot/loader.conf: >=20 > vm.kmem_size=3D"8192M" > vfs.zfs.arc_max=3D"6144M" >=20 > kstat.zfs.misc.arcstats.size tops out at around 6240986896, with Wired > in top(1) showing ~6.3GB. This is expected behaviour and fits (I > think) what people expect. >=20 > However, on the exact same system with these two settings: >=20 > vm.kmem_size=3D"16384M" > vfs.zfs.arc_max=3D"6144M" >=20 > The above ARC numbers are exactly *half* that amount. This is easily > reproducible. >=20 > Can someone 1) justify the "2x the amount of RAM for vm.kmem_size" > setting, and 2) explain in detail the above behaviour? >=20 > --=20 > | Jeremy Chadwick jdc@parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP 4BD6C0CB | >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" I guess Pawel can: vm.kmem_size=3D=E2=80=9D6G=E2=80=9D # This should be 150% of your RAM. http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-confi= guration/