Date: Sat, 27 Sep 2014 14:24:45 -0400 From: FF <fusionfoto@gmail.com> To: freebsd-fs@freebsd.org Subject: Re: Unexpected zfs ARC behavior Message-ID: <CAD=tpeejUezjRWh17u_w7HcdzqgVWc768p3FcsCTheeoOrduPQ@mail.gmail.com> In-Reply-To: <CAD=tpeeibpFb542R-atN=v6qwyOBguKhT2AtevTqqwXR0J4iwA@mail.gmail.com> References: <CAD=tpeeibpFb542R-atN=v6qwyOBguKhT2AtevTqqwXR0J4iwA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, forwarding from -questions. It looks like the image didn't make it across, so here it is: http://snag.gy/Ghb7X.jpg Thanks in advance for any pointers or suggestions, or confirmations that somehow this behavior is normal. -- So on a somewhat loaded ZFS file server (all NFS) serving mostly VMs with the load steadily increasing over the month (migrated from other servers)... the ARC size has unexpectedly dropped (please see attached graphic, if it makes through the mailing list server). The system peaks around 2,000 NFS IOPS and hasn't exhibited any slow downs. The L2ARC has a very low hit rate and prefetch has been turned off to increase the L2ARC efficiency... but none of that should really matter as far as I can tell since the L1ARC should try to use all the memory it can. > > Some tuning of zfs sysctls (mostly write_boost and write_max) to increase > them, but this has since been backed off to default. > > vmstat -m reports that 24G is dedicated to opensolaris: > solaris 1113433 24565820K - 6185559839 > 16,32,64,128,256,512,1024,2048,4096 > > And top reports memory free: > > last pid: 32075; load averages: 0.16, 0.13, > 0.14 up 36+21:44:59 09:16:49 > 24 processes: 1 running, 23 sleeping > CPU: 0.0% user, 0.0% nice, 0.0% system, 0.6% interrupt, 99.4% idle > Mem: 25M Active, 1187M Inact, 26G Wired, 2048K Cache, 1536M Buf, 4160M Free > ARC: 14G Total, 1679M MFU, 12G MRU, 28M Anon, 156M Header, 21M Other > > zfs-stats -a : > > ------------------------------------------------------------------------ > ZFS Subsystem Report Sat Sep 27 09:17:07 2014 > ------------------------------------------------------------------------ > > System Information: > > Kernel Version: 902001 (osreldate) > Hardware Platform: amd64 > Processor Architecture: amd64 > > ZFS Storage pool Version: 5000 > ZFS Filesystem Version: 5 > > FreeBSD 9.2-RELEASE-p10 #0 r270148M: Mon Aug 18 23:14:36 EDT 2014 root > 9:17AM up 36 days, 21:45, 2 users, load averages: 0.12, 0.12, 0.13 > > ------------------------------------------------------------------------ > > System Memory: > > 0.08% 26.22 MiB Active, 3.75% 1.16 GiB Inact > 82.36% 25.47 GiB Wired, 0.01% 2.00 MiB Cache > 13.80% 4.27 GiB Free, 0.00% 1.03 MiB Gap > > Real Installed: 32.00 GiB > Real Available: 99.63% 31.88 GiB > Real Managed: 97.01% 30.93 GiB > > Logical Total: 32.00 GiB > Logical Used: 83.03% 26.57 GiB > Logical Free: 16.97% 5.43 GiB > > Kernel Memory: 23.54 GiB > Data: 99.90% 23.52 GiB > Text: 0.10% 23.13 MiB > > Kernel Memory Map: 29.76 GiB > Size: 76.40% 22.74 GiB > Free: 23.60% 7.02 GiB > > ------------------------------------------------------------------------ > > ARC Summary: (HEALTHY) > Memory Throttle Count: 0 > > ARC Misc: > Deleted: 90.09m > Recycle Misses: 2.44m > Mutex Misses: 794.67k > Evict Skips: 17.90m > > ARC Size: 44.78% 13.40 GiB > Target Size: (Adaptive) 44.78% 13.40 GiB > Min Size (Hard Limit): 12.50% 3.74 GiB > Max Size (High Water): 8:1 29.93 GiB > > ARC Size Breakdown: > Recently Used Cache Size: 86.16% 11.55 GiB > Frequently Used Cache Size: 13.84% 1.85 GiB > > ARC Hash Breakdown: > Elements Max: 786.71k > Elements Current: 87.05% 684.85k > Collisions: 153.35m > Chain Max: 16 > Chains: 194.92k > > ------------------------------------------------------------------------ > > ARC Efficiency: 506.24m > Cache Hit Ratio: 87.56% 443.25m > Cache Miss Ratio: 12.44% 62.99m > Actual Hit Ratio: 80.06% 405.29m > > Data Demand Efficiency: 93.92% 372.74m > Data Prefetch Efficiency: 49.49% 69.76m > > CACHE HITS BY CACHE LIST: > Anonymously Used: 6.10% 27.05m > Most Recently Used: 31.37% 139.05m > Most Frequently Used: 60.07% 266.24m > Most Recently Used Ghost: 0.80% 3.56m > Most Frequently Used Ghost: 1.66% 7.35m > > CACHE HITS BY DATA TYPE: > Demand Data: 78.98% 350.09m > Prefetch Data: 7.79% 34.53m > Demand Metadata: 12.15% 53.86m > Prefetch Metadata: 1.08% 4.77m > > CACHE MISSES BY DATA TYPE: > Demand Data: 35.95% 22.65m > Prefetch Data: 55.93% 35.23m > Demand Metadata: 4.49% 2.83m > Prefetch Metadata: 3.63% 2.29m > > ------------------------------------------------------------------------ > <cut> > > Any suggestions? Is this expected or acceptable behavior? > > Thanks in advance, > > -- > FF > -- FF
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD=tpeejUezjRWh17u_w7HcdzqgVWc768p3FcsCTheeoOrduPQ>