Date: Sat, 27 Sep 2014 09:20:31 -0400 From: FF <fusionfoto@gmail.com> To: freebsd-questions@freebsd.org Subject: Unexpected zfs ARC behavior Message-ID: <CAD=tpeeibpFb542R-atN=v6qwyOBguKhT2AtevTqqwXR0J4iwA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
So on a somewhat loaded ZFS file server (all NFS) serving mostly VMs with
the load steadily increasing over the month (migrated from other
servers)... the ARC size has unexpectedly dropped (please see attached
graphic, if it makes through the mailing list server). The system peaks
around 2,000 NFS IOPS and hasn't exhibited any slow downs. The L2ARC has a
very low hit rate and prefetch has been turned off to increase the L2ARC
efficiency... but none of that should really matter as far as I can tell
since the L1ARC should try to use all the memory it can.
Some tuning of zfs sysctls (mostly write_boost and write_max) to increase
them, but this has since been backed off to default.
vmstat -m reports that 24G is dedicated to opensolaris:
solaris 1113433 24565820K - 6185559839
16,32,64,128,256,512,1024,2048,4096
And top reports memory free:
last pid: 32075; load averages: 0.16, 0.13,
0.14 up 36+21:44:59 09:16:49
24 processes: 1 running, 23 sleeping
CPU: 0.0% user, 0.0% nice, 0.0% system, 0.6% interrupt, 99.4% idle
Mem: 25M Active, 1187M Inact, 26G Wired, 2048K Cache, 1536M Buf, 4160M Free
ARC: 14G Total, 1679M MFU, 12G MRU, 28M Anon, 156M Header, 21M Other
zfs-stats -a :
------------------------------------------------------------------------
ZFS Subsystem Report Sat Sep 27 09:17:07 2014
------------------------------------------------------------------------
System Information:
Kernel Version: 902001 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 5000
ZFS Filesystem Version: 5
FreeBSD 9.2-RELEASE-p10 #0 r270148M: Mon Aug 18 23:14:36 EDT 2014 root
9:17AM up 36 days, 21:45, 2 users, load averages: 0.12, 0.12, 0.13
------------------------------------------------------------------------
System Memory:
0.08% 26.22 MiB Active, 3.75% 1.16 GiB Inact
82.36% 25.47 GiB Wired, 0.01% 2.00 MiB Cache
13.80% 4.27 GiB Free, 0.00% 1.03 MiB Gap
Real Installed: 32.00 GiB
Real Available: 99.63% 31.88 GiB
Real Managed: 97.01% 30.93 GiB
Logical Total: 32.00 GiB
Logical Used: 83.03% 26.57 GiB
Logical Free: 16.97% 5.43 GiB
Kernel Memory: 23.54 GiB
Data: 99.90% 23.52 GiB
Text: 0.10% 23.13 MiB
Kernel Memory Map: 29.76 GiB
Size: 76.40% 22.74 GiB
Free: 23.60% 7.02 GiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 90.09m
Recycle Misses: 2.44m
Mutex Misses: 794.67k
Evict Skips: 17.90m
ARC Size: 44.78% 13.40 GiB
Target Size: (Adaptive) 44.78% 13.40 GiB
Min Size (Hard Limit): 12.50% 3.74 GiB
Max Size (High Water): 8:1 29.93 GiB
ARC Size Breakdown:
Recently Used Cache Size: 86.16% 11.55 GiB
Frequently Used Cache Size: 13.84% 1.85 GiB
ARC Hash Breakdown:
Elements Max: 786.71k
Elements Current: 87.05% 684.85k
Collisions: 153.35m
Chain Max: 16
Chains: 194.92k
------------------------------------------------------------------------
ARC Efficiency: 506.24m
Cache Hit Ratio: 87.56% 443.25m
Cache Miss Ratio: 12.44% 62.99m
Actual Hit Ratio: 80.06% 405.29m
Data Demand Efficiency: 93.92% 372.74m
Data Prefetch Efficiency: 49.49% 69.76m
CACHE HITS BY CACHE LIST:
Anonymously Used: 6.10% 27.05m
Most Recently Used: 31.37% 139.05m
Most Frequently Used: 60.07% 266.24m
Most Recently Used Ghost: 0.80% 3.56m
Most Frequently Used Ghost: 1.66% 7.35m
CACHE HITS BY DATA TYPE:
Demand Data: 78.98% 350.09m
Prefetch Data: 7.79% 34.53m
Demand Metadata: 12.15% 53.86m
Prefetch Metadata: 1.08% 4.77m
CACHE MISSES BY DATA TYPE:
Demand Data: 35.95% 22.65m
Prefetch Data: 55.93% 35.23m
Demand Metadata: 4.49% 2.83m
Prefetch Metadata: 3.63% 2.29m
------------------------------------------------------------------------
<cut>
Any suggestions? Is this expected or acceptable behavior?
Thanks in advance,
--
FF
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD=tpeeibpFb542R-atN=v6qwyOBguKhT2AtevTqqwXR0J4iwA>