From owner-freebsd-stable@FreeBSD.ORG Wed Nov 5 02:31:50 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9D217C07 for ; Wed, 5 Nov 2014 02:31:50 +0000 (UTC) Received: from mail-pa0-x233.google.com (mail-pa0-x233.google.com [IPv6:2607:f8b0:400e:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 70C75C03 for ; Wed, 5 Nov 2014 02:31:50 +0000 (UTC) Received: by mail-pa0-f51.google.com with SMTP id kq14so15603517pab.24 for ; Tue, 04 Nov 2014 18:31:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=BJ4AydYoWBIuD4jewSaOaPUS+9tMsWjV8QRVot/6Vx8=; b=t2h/h1CYpy4e5Bqi2DdJIGUFiuoNjbwkpC8VczIcYnVZbqUh+UzPPaNWUW/1XwvHDY E+8UakxIJEyrIVtBehm8c6FN4Yvo9nBKPrT9zeWpidmnMIT9HPPAe362sghHjWX3VUNx 1g1h2ESLxNfFiKb3x05fbUV3duKkIMZg64KnG/imTZH9D57XfBzKN+rCeRWrMknccUbH GsT7SGs1gyblBOJld/6ttzYmsNMvgKdUQrYGk+SqMX6xMot2cTdca9dmSMuCongHDiSQ dTOg3bAuFVP/oBW5Tjy6Dwmw1/pKg9HhvsBqrIDWpNL3WPdEoBFkWzzyxfsZ+wBHH/tP aysg== X-Received: by 10.66.119.175 with SMTP id kv15mr2491388pab.30.1415154710021; Tue, 04 Nov 2014 18:31:50 -0800 (PST) Received: from [10.51.2.189] ([199.101.130.202]) by mx.google.com with ESMTPSA id je2sm1517727pbd.94.2014.11.04.18.31.47 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Nov 2014 18:31:48 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1990.1\)) Subject: Re: ARC size limit From: George Kola In-Reply-To: <5459118B.7090904@multiplay.co.uk> Date: Tue, 4 Nov 2014 18:31:47 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <34588CE4-F44A-4D6D-BB38-28D55BE95D4C@gmail.com> References: <3B70EA0C-0976-49D6-8418-6B5D22ED7E65@gmail.com> <54589722.3080803@multiplay.co.uk> <5459118B.7090904@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.1990.1) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2014 02:31:50 -0000 Thanks for the patch Steven. We are now running it. Thanks, George > On Nov 4, 2014, at 9:48 AM, Steven Hartland = wrote: >=20 > Try the attached, its the patch we'll be using when we roll out 10.1. >=20 > 1. Apply with: cd /usr/src && patch < zfs-arc-refactor.patch > 2. Rebuild your kernel > 3. Install the new kernel > 4. Reboot >=20 > Regards > Steve > On 04/11/2014 16:48, George Kola wrote: >> Thanks Steven. We are running 10.1rc3. Is the best way to get this = patch to just compile 10-stable kernel ? We are new to running FreeBSD = and hence the newbie question. >>=20 >>=20 >> Thanks, >> George >>=20 >>=20 >>=20 >>> On Nov 4, 2014, at 1:06 AM, Steven Hartland = wrote: >>>=20 >>> You need = https://svnweb.freebsd.org/base?view=3Drevision&revision=3D272875 >>> On 04/11/2014 06:29, George Kola wrote: >>>> Hi All, >>>> This is my first post to freebsd-stable fresh of Meet BSD = California 2014. We are switching our entire production to FreeBSD. Our = storage servers have 256 GB of RAM , 4 TB of SSD and 40 TB of spinning = disks. We are running ZFS root and the SSD is configured as L2ARC. We = are running FreeBSD 10.1 RC3 >>>> I am finding that on all our machines, ARC is somehow = limited to < 64 GB of memory and we have a huge inactive memory (180 G). = The surprising thing is that ARC seems to have almost the same limit (< = 64 GB) on all of our storage boxes and ARC is not growing even though = L2ARC hit shows that there is advantage in growing ARC. >>>> Any help/pointers is appreciated. >>>> What I am trying to do is to tune ZFS for our workload. We = are hoping that we get a high hit rate. >>>> Thanks to Justin Gibbs and Allan Jude for initial pointers = and help. They suggested posting to the mailing list to get further = help. >>>>=20 >>>> I have pasted top output and zfs-stats output below and yes = UMA is enabled. >>>>=20 >>>> Thanks, >>>> George >>>> =20 >>>> top >>>> last pid: 27458; load averages: 3.30, 5.42, 5.34 = = up 6+09:59:30 05:38:49 >>>> 71 processes: 1 running, 70 sleeping >>>> CPU: 4.2% user, 0.0% nice, 4.6% system, 0.2% interrupt, 90.9% = idle >>>> Mem: 11G Active, 181G Inact, 52G Wired, 1368M Cache, 4266M Free >>>> ARC: 47G Total, 1555M MFU, 41G MRU, 35M Anon, 3984M Header, 709M = Other >>>> Swap: 64G Total, 2874M Used, 61G Free, 4% Inuse >>>>=20 >>>>=20 >>>>=20 >>>> sysctl vfs.zfs.zio.use_uma >>>> vfs.zfs.zio.use_uma: 1 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> zfs-mon -a output >>>>=20 >>>> ZFS real-time cache activity monitor >>>> Seconds elapsed: 62 >>>>=20 >>>> Cache hits and misses: >>>> 1s 10s 60s tot >>>> ARC hits: 124 126 103 101 >>>> ARC misses: 35 46 29 28 >>>> ARC demand data hits: 55 90 61 61 >>>> ARC demand data misses: 20 32 18 17 >>>> ARC demand metadata hits: 69 36 42 40 >>>> ARC demand metadata misses: 9 13 10 9 >>>> ARC prefetch data hits: 0 0 0 0 >>>> ARC prefetch data misses: 6 1 1 1 >>>> ARC prefetch metadata hits: 0 0 0 0 >>>> ARC prefetch metadata misses: 0 0 0 0 >>>> L2ARC hits: 16 28 14 14 >>>> L2ARC misses: 19 18 15 14 >>>> ZFETCH hits: 592 2842 2098 2047 >>>> ZFETCH misses: 308 1326 507 494 >>>>=20 >>>> Cache efficiency percentage: >>>> 10s 60s tot >>>> ARC: 73.26 78.03 78.29 >>>> ARC demand data: 73.77 77.22 78.21 >>>> ARC demand metadata: 73.47 80.77 81.63 >>>> ARC prefetch data: 0.00 0.00 0.00 >>>> ARC prefetch metadata: 0.00 0.00 0.00 >>>> L2ARC: 60.87 48.28 50.00 >>>> ZFETCH: 68.19 80.54 80.56 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> zfs-stats -a output >>>>=20 >>>> ZFS real-time cache activity monitor >>>> Seconds elapsed: 62 >>>>=20 >>>> Cache hits and misses: >>>> 1s 10s 60s tot >>>> ARC hits: 124 126 103 101 >>>> ARC misses: 35 46 29 28 >>>> ARC demand data hits: 55 90 61 61 >>>> ARC demand data misses: 20 32 18 17 >>>> ARC demand metadata hits: 69 36 42 40 >>>> ARC demand metadata misses: 9 13 10 9 >>>> ARC prefetch data hits: 0 0 0 0 >>>> ARC prefetch data misses: 6 1 1 1 >>>> ARC prefetch metadata hits: 0 0 0 0 >>>> ARC prefetch metadata misses: 0 0 0 0 >>>> L2ARC hits: 16 28 14 14 >>>> L2ARC misses: 19 18 15 14 >>>> ZFETCH hits: 592 2842 2098 2047 >>>> ZFETCH misses: 308 1326 507 494 >>>>=20 >>>> Cache efficiency percentage: >>>> 10s 60s tot >>>> ARC: 73.26 78.03 78.29 >>>> ARC demand data: 73.77 77.22 78.21 >>>> ARC demand metadata: 73.47 80.77 81.63 >>>> ARC prefetch data: 0.00 0.00 0.00 >>>> ARC prefetch metadata: 0.00 0.00 0.00 >>>> L2ARC: 60.87 48.28 50.00 >>>> ZFETCH: 68.19 80.54 80.56 >>>>=20 >>>>=20 >>>>=20 >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 >