From owner-freebsd-fs@FreeBSD.ORG Sat Sep 27 18:24:46 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A442A1E2 for ; Sat, 27 Sep 2014 18:24:46 +0000 (UTC) Received: from mail-vc0-x22c.google.com (mail-vc0-x22c.google.com [IPv6:2607:f8b0:400c:c03::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 62AAFE2B for ; Sat, 27 Sep 2014 18:24:46 +0000 (UTC) Received: by mail-vc0-f172.google.com with SMTP id lf12so184999vcb.3 for ; Sat, 27 Sep 2014 11:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=IFUrF0HdXyB6BJOc44SzX0RWgrMcP8FdfiRrGi9SWXc=; b=oFMP8N6gN/F3CA4ZBsoWfex3rVrtUNuNnRZ0qJ95nJhHs96lFGHnK66qLzO+4uImSg f0xtPxPgCwmLHO4hqCzyxMDi0mVs5WVt0N1loNH7gPG19W3a/0sX+mWt0bbCtfCqgiPZ s6IIpFr3lXaEVe/HzVShT73P3gRi7lhpjwgb70/frm9oK+tV3kymh0N3zYHytXy57LjP okjUyJDeB9FHbA7vWecpFgMcRcBLT8M79GWwzzbadcdIDCgIKXZtMCm8z7wT6aD6knEO G3fvavZ8s3rWQ15jpd/zWSNDkd1cf2fkRU+UQOXe4s1Zh/UMNUkwA+hXSITjcGKTpvLu 2wOQ== MIME-Version: 1.0 X-Received: by 10.220.20.5 with SMTP id d5mr21754922vcb.9.1411842285236; Sat, 27 Sep 2014 11:24:45 -0700 (PDT) Received: by 10.31.160.7 with HTTP; Sat, 27 Sep 2014 11:24:45 -0700 (PDT) In-Reply-To: References: Date: Sat, 27 Sep 2014 14:24:45 -0400 Message-ID: Subject: Re: Unexpected zfs ARC behavior From: FF To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 18:24:46 -0000 Hi, forwarding from -questions. It looks like the image didn't make it across, so here it is: http://snag.gy/Ghb7X.jpg Thanks in advance for any pointers or suggestions, or confirmations that somehow this behavior is normal. -- So on a somewhat loaded ZFS file server (all NFS) serving mostly VMs with the load steadily increasing over the month (migrated from other servers)... the ARC size has unexpectedly dropped (please see attached graphic, if it makes through the mailing list server). The system peaks around 2,000 NFS IOPS and hasn't exhibited any slow downs. The L2ARC has a very low hit rate and prefetch has been turned off to increase the L2ARC efficiency... but none of that should really matter as far as I can tell since the L1ARC should try to use all the memory it can. > > Some tuning of zfs sysctls (mostly write_boost and write_max) to increase > them, but this has since been backed off to default. > > vmstat -m reports that 24G is dedicated to opensolaris: > solaris 1113433 24565820K - 6185559839 > 16,32,64,128,256,512,1024,2048,4096 > > And top reports memory free: > > last pid: 32075; load averages: 0.16, 0.13, > 0.14 up 36+21:44:59 09:16:49 > 24 processes: 1 running, 23 sleeping > CPU: 0.0% user, 0.0% nice, 0.0% system, 0.6% interrupt, 99.4% idle > Mem: 25M Active, 1187M Inact, 26G Wired, 2048K Cache, 1536M Buf, 4160M Free > ARC: 14G Total, 1679M MFU, 12G MRU, 28M Anon, 156M Header, 21M Other > > zfs-stats -a : > > ------------------------------------------------------------------------ > ZFS Subsystem Report Sat Sep 27 09:17:07 2014 > ------------------------------------------------------------------------ > > System Information: > > Kernel Version: 902001 (osreldate) > Hardware Platform: amd64 > Processor Architecture: amd64 > > ZFS Storage pool Version: 5000 > ZFS Filesystem Version: 5 > > FreeBSD 9.2-RELEASE-p10 #0 r270148M: Mon Aug 18 23:14:36 EDT 2014 root > 9:17AM up 36 days, 21:45, 2 users, load averages: 0.12, 0.12, 0.13 > > ------------------------------------------------------------------------ > > System Memory: > > 0.08% 26.22 MiB Active, 3.75% 1.16 GiB Inact > 82.36% 25.47 GiB Wired, 0.01% 2.00 MiB Cache > 13.80% 4.27 GiB Free, 0.00% 1.03 MiB Gap > > Real Installed: 32.00 GiB > Real Available: 99.63% 31.88 GiB > Real Managed: 97.01% 30.93 GiB > > Logical Total: 32.00 GiB > Logical Used: 83.03% 26.57 GiB > Logical Free: 16.97% 5.43 GiB > > Kernel Memory: 23.54 GiB > Data: 99.90% 23.52 GiB > Text: 0.10% 23.13 MiB > > Kernel Memory Map: 29.76 GiB > Size: 76.40% 22.74 GiB > Free: 23.60% 7.02 GiB > > ------------------------------------------------------------------------ > > ARC Summary: (HEALTHY) > Memory Throttle Count: 0 > > ARC Misc: > Deleted: 90.09m > Recycle Misses: 2.44m > Mutex Misses: 794.67k > Evict Skips: 17.90m > > ARC Size: 44.78% 13.40 GiB > Target Size: (Adaptive) 44.78% 13.40 GiB > Min Size (Hard Limit): 12.50% 3.74 GiB > Max Size (High Water): 8:1 29.93 GiB > > ARC Size Breakdown: > Recently Used Cache Size: 86.16% 11.55 GiB > Frequently Used Cache Size: 13.84% 1.85 GiB > > ARC Hash Breakdown: > Elements Max: 786.71k > Elements Current: 87.05% 684.85k > Collisions: 153.35m > Chain Max: 16 > Chains: 194.92k > > ------------------------------------------------------------------------ > > ARC Efficiency: 506.24m > Cache Hit Ratio: 87.56% 443.25m > Cache Miss Ratio: 12.44% 62.99m > Actual Hit Ratio: 80.06% 405.29m > > Data Demand Efficiency: 93.92% 372.74m > Data Prefetch Efficiency: 49.49% 69.76m > > CACHE HITS BY CACHE LIST: > Anonymously Used: 6.10% 27.05m > Most Recently Used: 31.37% 139.05m > Most Frequently Used: 60.07% 266.24m > Most Recently Used Ghost: 0.80% 3.56m > Most Frequently Used Ghost: 1.66% 7.35m > > CACHE HITS BY DATA TYPE: > Demand Data: 78.98% 350.09m > Prefetch Data: 7.79% 34.53m > Demand Metadata: 12.15% 53.86m > Prefetch Metadata: 1.08% 4.77m > > CACHE MISSES BY DATA TYPE: > Demand Data: 35.95% 22.65m > Prefetch Data: 55.93% 35.23m > Demand Metadata: 4.49% 2.83m > Prefetch Metadata: 3.63% 2.29m > > ------------------------------------------------------------------------ > > > Any suggestions? Is this expected or acceptable behavior? > > Thanks in advance, > > -- > FF > -- FF