Date: Fri, 4 Mar 2011 06:38:01 -0800 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Adam McDougall <mcdouga9@egr.msu.edu> Cc: freebsd-fs@freebsd.org Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE Message-ID: <20110304143801.GA1734@icarus.home.lan> In-Reply-To: <4D70E7D1.5030205@egr.msu.edu> References: <1299232133.18671.3.camel@pc286.embl.fr> <4D70E7D1.5030205@egr.msu.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Mar 04, 2011 at 08:23:29AM -0500, Adam McDougall wrote: > On 03/04/11 04:48, Mickaël Canévet wrote: > >>I'd use vm.kmem_size="32G" (i.e. twice your RAM) and that's it. > > > >Should I also increase vfs.zfs.arc_max ? > > > >Do you have any idea why the kernel panicked at only 8GB allocated ? > > > >Thank you > > I believe ARC allocations in kmem can become fragmented, so when it > is searching for a place to store a new contiguous segment of > memory, the remaining fragmented free spaces may all be too small. > I also set > vm.kmem_size to about twice the amount of ram to help it avoid this > issue. I suspect if kmem is badly fragmented then performance of > ZFS can downgrade, so that is another reason to keep kmem bigger. My findings on 8.2-RELEASE indicate that doing this results in very unexpected behaviour regarding the ARC maximum. As such, I cannot recommend this model. For example, on an amd64 system with 8GB physical RAM and these two settings in /boot/loader.conf: vm.kmem_size="8192M" vfs.zfs.arc_max="6144M" kstat.zfs.misc.arcstats.size tops out at around 6240986896, with Wired in top(1) showing ~6.3GB. This is expected behaviour and fits (I think) what people expect. However, on the exact same system with these two settings: vm.kmem_size="16384M" vfs.zfs.arc_max="6144M" The above ARC numbers are exactly *half* that amount. This is easily reproducible. Can someone 1) justify the "2x the amount of RAM for vm.kmem_size" setting, and 2) explain in detail the above behaviour? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110304143801.GA1734>