From owner-freebsd-questions@FreeBSD.ORG Sat Sep 27 13:20:32 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9F5C22AE for ; Sat, 27 Sep 2014 13:20:32 +0000 (UTC) Received: from mail-pd0-x235.google.com (mail-pd0-x235.google.com [IPv6:2607:f8b0:400e:c02::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 75339D7A for ; Sat, 27 Sep 2014 13:20:32 +0000 (UTC) Received: by mail-pd0-f181.google.com with SMTP id z10so601890pdj.26 for ; Sat, 27 Sep 2014 06:20:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=0tl5zy8kH0F8LxAzDO/uoQZVLfm+DIexMsrHAfgO8vQ=; b=ikeAfhL6DBxUYYdJ6W10+eyVB8rpwdw4YrYB0SBd6tkCHEOK3VbYsFuGaGSVRKJlVY KUXhEpgYnf6/hMPkdDoFBNvQKYowwWu+puDZvvOPZLdfMZLksCsB07vYQXwwb9lzE8W3 JiHlNXZHnyrLjhAD1xzOTctQ3PU5tAnz9yWoY8SStd6mjMiENi0V9Xv34vIFNIK/nIyd 03JckhgkZ2n67/XSrO5eaPf3E5FI/IugrovNcKqFCsK47YnYErzhSwi2YxEbphiFsFSB d3a2LExfo1z1x9zSENc2CQZEhniI/PksQaG2O4Wn9FNF4MMvM8VcTUiUKUaxFxHKYa99 je4A== MIME-Version: 1.0 X-Received: by 10.70.6.162 with SMTP id c2mr26447574pda.2.1411824031743; Sat, 27 Sep 2014 06:20:31 -0700 (PDT) Received: by 10.70.128.40 with HTTP; Sat, 27 Sep 2014 06:20:31 -0700 (PDT) Date: Sat, 27 Sep 2014 09:20:31 -0400 Message-ID: Subject: Unexpected zfs ARC behavior From: FF To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 13:20:32 -0000 So on a somewhat loaded ZFS file server (all NFS) serving mostly VMs with the load steadily increasing over the month (migrated from other servers)... the ARC size has unexpectedly dropped (please see attached graphic, if it makes through the mailing list server). The system peaks around 2,000 NFS IOPS and hasn't exhibited any slow downs. The L2ARC has a very low hit rate and prefetch has been turned off to increase the L2ARC efficiency... but none of that should really matter as far as I can tell since the L1ARC should try to use all the memory it can. Some tuning of zfs sysctls (mostly write_boost and write_max) to increase them, but this has since been backed off to default. vmstat -m reports that 24G is dedicated to opensolaris: solaris 1113433 24565820K - 6185559839 16,32,64,128,256,512,1024,2048,4096 And top reports memory free: last pid: 32075; load averages: 0.16, 0.13, 0.14 up 36+21:44:59 09:16:49 24 processes: 1 running, 23 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.6% interrupt, 99.4% idle Mem: 25M Active, 1187M Inact, 26G Wired, 2048K Cache, 1536M Buf, 4160M Free ARC: 14G Total, 1679M MFU, 12G MRU, 28M Anon, 156M Header, 21M Other zfs-stats -a : ------------------------------------------------------------------------ ZFS Subsystem Report Sat Sep 27 09:17:07 2014 ------------------------------------------------------------------------ System Information: Kernel Version: 902001 (osreldate) Hardware Platform: amd64 Processor Architecture: amd64 ZFS Storage pool Version: 5000 ZFS Filesystem Version: 5 FreeBSD 9.2-RELEASE-p10 #0 r270148M: Mon Aug 18 23:14:36 EDT 2014 root 9:17AM up 36 days, 21:45, 2 users, load averages: 0.12, 0.12, 0.13 ------------------------------------------------------------------------ System Memory: 0.08% 26.22 MiB Active, 3.75% 1.16 GiB Inact 82.36% 25.47 GiB Wired, 0.01% 2.00 MiB Cache 13.80% 4.27 GiB Free, 0.00% 1.03 MiB Gap Real Installed: 32.00 GiB Real Available: 99.63% 31.88 GiB Real Managed: 97.01% 30.93 GiB Logical Total: 32.00 GiB Logical Used: 83.03% 26.57 GiB Logical Free: 16.97% 5.43 GiB Kernel Memory: 23.54 GiB Data: 99.90% 23.52 GiB Text: 0.10% 23.13 MiB Kernel Memory Map: 29.76 GiB Size: 76.40% 22.74 GiB Free: 23.60% 7.02 GiB ------------------------------------------------------------------------ ARC Summary: (HEALTHY) Memory Throttle Count: 0 ARC Misc: Deleted: 90.09m Recycle Misses: 2.44m Mutex Misses: 794.67k Evict Skips: 17.90m ARC Size: 44.78% 13.40 GiB Target Size: (Adaptive) 44.78% 13.40 GiB Min Size (Hard Limit): 12.50% 3.74 GiB Max Size (High Water): 8:1 29.93 GiB ARC Size Breakdown: Recently Used Cache Size: 86.16% 11.55 GiB Frequently Used Cache Size: 13.84% 1.85 GiB ARC Hash Breakdown: Elements Max: 786.71k Elements Current: 87.05% 684.85k Collisions: 153.35m Chain Max: 16 Chains: 194.92k ------------------------------------------------------------------------ ARC Efficiency: 506.24m Cache Hit Ratio: 87.56% 443.25m Cache Miss Ratio: 12.44% 62.99m Actual Hit Ratio: 80.06% 405.29m Data Demand Efficiency: 93.92% 372.74m Data Prefetch Efficiency: 49.49% 69.76m CACHE HITS BY CACHE LIST: Anonymously Used: 6.10% 27.05m Most Recently Used: 31.37% 139.05m Most Frequently Used: 60.07% 266.24m Most Recently Used Ghost: 0.80% 3.56m Most Frequently Used Ghost: 1.66% 7.35m CACHE HITS BY DATA TYPE: Demand Data: 78.98% 350.09m Prefetch Data: 7.79% 34.53m Demand Metadata: 12.15% 53.86m Prefetch Metadata: 1.08% 4.77m CACHE MISSES BY DATA TYPE: Demand Data: 35.95% 22.65m Prefetch Data: 55.93% 35.23m Demand Metadata: 4.49% 2.83m Prefetch Metadata: 3.63% 2.29m ------------------------------------------------------------------------ Any suggestions? Is this expected or acceptable behavior? Thanks in advance, -- FF