From owner-freebsd-stable@FreeBSD.ORG Tue Nov 4 06:29:53 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 02624AD3; Tue, 4 Nov 2014 06:29:53 +0000 (UTC) Received: from mail-pd0-x231.google.com (mail-pd0-x231.google.com [IPv6:2607:f8b0:400e:c02::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB01288B; Tue, 4 Nov 2014 06:29:52 +0000 (UTC) Received: by mail-pd0-f177.google.com with SMTP id v10so12850160pde.8 for ; Mon, 03 Nov 2014 22:29:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:date:message-id:cc:to:mime-version; bh=JWH5EQ1my4Wz6RJXvJ721yy7b87WiYUpe2OSpn/3O4I=; b=fvevAymDOP+NgbmkIJaWuOwwZga9tjc+JvLwclazJbSJUBl/2pnfZ2AAWF5Ha+US/g Oq62ijOdnvsN0OD6l9GZkD3u9ZWZbblWKYJ6OjWyU1YwZpNPfllkuqOioVPuw45knyTK dmvrF/pcHg4FaN1oCesFrTtyxxEtUf6Xzv+5PWH7UcRr75JSemxflxJswYcgj/4LcnDp TmMiw1r6I4CbYVCrRVnvaeg5xmQbHuw+YhAWHAZ6oNGUv4F9KE3VyaOf28g/M8MhzVby nmnQ1NLB7OcyB/mzP3fXCwJPZKhC6QqWiELWn7lpOTt2Ddq8U4N4Rrf/I3UEuJjNZvey eChw== X-Received: by 10.68.232.226 with SMTP id tr2mr2050645pbc.89.1415082592274; Mon, 03 Nov 2014 22:29:52 -0800 (PST) Received: from [10.0.1.8] (50-0-30-197.dsl.dynamic.fusionbroadband.com. [50.0.30.197]) by mx.google.com with ESMTPSA id kc15sm3310624pbb.3.2014.11.03.22.29.50 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Nov 2014 22:29:51 -0800 (PST) From: George Kola Subject: ARC size limit Date: Mon, 3 Nov 2014 22:29:48 -0800 Message-Id: <3B70EA0C-0976-49D6-8418-6B5D22ED7E65@gmail.com> To: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1990.1\)) X-Mailer: Apple Mail (2.1990.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: alc@FreeBSD.org, gibbs@FreeBSD.org, kib@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 06:29:53 -0000 Hi All, This is my first post to freebsd-stable fresh of Meet BSD = California 2014. We are switching our entire production to FreeBSD. Our = storage servers have 256 GB of RAM , 4 TB of SSD and 40 TB of spinning = disks. We are running ZFS root and the SSD is configured as L2ARC. We = are running FreeBSD 10.1 RC3 I am finding that on all our machines, ARC is somehow limited to = < 64 GB of memory and we have a huge inactive memory (180 G). The = surprising thing is that ARC seems to have almost the same limit (< 64 = GB) on all of our storage boxes and ARC is not growing even though L2ARC = hit shows that there is advantage in growing ARC. Any help/pointers is appreciated. What I am trying to do is to tune ZFS for our workload. We are = hoping that we get a high hit rate.=20 Thanks to Justin Gibbs and Allan Jude for initial pointers and = help. They suggested posting to the mailing list to get further help. I have pasted top output and zfs-stats output below and yes UMA = is enabled. Thanks, George =20 top last pid: 27458; load averages: 3.30, 5.42, 5.34 = = up 6+09:59:30 05:38:49 71 processes: 1 running, 70 sleeping CPU: 4.2% user, 0.0% nice, 4.6% system, 0.2% interrupt, 90.9% idle Mem: 11G Active, 181G Inact, 52G Wired, 1368M Cache, 4266M Free ARC: 47G Total, 1555M MFU, 41G MRU, 35M Anon, 3984M Header, 709M Other Swap: 64G Total, 2874M Used, 61G Free, 4% Inuse sysctl vfs.zfs.zio.use_uma vfs.zfs.zio.use_uma: 1 zfs-mon -a output ZFS real-time cache activity monitor Seconds elapsed: 62 Cache hits and misses: 1s 10s 60s tot ARC hits: 124 126 103 101 ARC misses: 35 46 29 28 ARC demand data hits: 55 90 61 61 ARC demand data misses: 20 32 18 17 ARC demand metadata hits: 69 36 42 40 ARC demand metadata misses: 9 13 10 9 ARC prefetch data hits: 0 0 0 0 ARC prefetch data misses: 6 1 1 1 ARC prefetch metadata hits: 0 0 0 0 ARC prefetch metadata misses: 0 0 0 0 L2ARC hits: 16 28 14 14 L2ARC misses: 19 18 15 14 ZFETCH hits: 592 2842 2098 2047 ZFETCH misses: 308 1326 507 494 Cache efficiency percentage: 10s 60s tot ARC: 73.26 78.03 78.29 ARC demand data: 73.77 77.22 78.21 ARC demand metadata: 73.47 80.77 81.63 ARC prefetch data: 0.00 0.00 0.00 ARC prefetch metadata: 0.00 0.00 0.00 L2ARC: 60.87 48.28 50.00 ZFETCH: 68.19 80.54 80.56 zfs-stats -a output ZFS real-time cache activity monitor Seconds elapsed: 62 Cache hits and misses: 1s 10s 60s tot ARC hits: 124 126 103 101 ARC misses: 35 46 29 28 ARC demand data hits: 55 90 61 61 ARC demand data misses: 20 32 18 17 ARC demand metadata hits: 69 36 42 40 ARC demand metadata misses: 9 13 10 9 ARC prefetch data hits: 0 0 0 0 ARC prefetch data misses: 6 1 1 1 ARC prefetch metadata hits: 0 0 0 0 ARC prefetch metadata misses: 0 0 0 0 L2ARC hits: 16 28 14 14 L2ARC misses: 19 18 15 14 ZFETCH hits: 592 2842 2098 2047 ZFETCH misses: 308 1326 507 494 Cache efficiency percentage: 10s 60s tot ARC: 73.26 78.03 78.29 ARC demand data: 73.77 77.22 78.21 ARC demand metadata: 73.47 80.77 81.63 ARC prefetch data: 0.00 0.00 0.00 ARC prefetch metadata: 0.00 0.00 0.00 L2ARC: 60.87 48.28 50.00 ZFETCH: 68.19 80.54 80.56