From owner-freebsd-stable@FreeBSD.ORG Tue Nov 4 16:53:40 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C7783513 for ; Tue, 4 Nov 2014 16:53:40 +0000 (UTC) Received: from mail-pd0-x22f.google.com (mail-pd0-x22f.google.com [IPv6:2607:f8b0:400e:c02::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9921CD87 for ; Tue, 4 Nov 2014 16:53:40 +0000 (UTC) Received: by mail-pd0-f175.google.com with SMTP id y13so13995323pdi.20 for ; Tue, 04 Nov 2014 08:53:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=9ikyDh86KZYAbcNkOfa+F6Qr4o4o503baFI8GIzU7rs=; b=ESoN0ud3FIn7Hwb8l0pzApv3gfEAeEP2GXW6qmaBgbttzSXtKcuhEM7Ke/Bhky1Xwf NAtdXubtsJiYD/3SjYFHBzhG4ZrrVeFcxN19hgLsMUVClLio/WO44PiRZU+/Rvbcb8jl YPI4RxCIwckEMS5VdoPoHlo5rIxqFv26qvWeavUm+esNUah4QDr8+856O5k5elDoyFC0 KX3FXkAHelA5QEVkBg49Y8cyB3rO/i1T3RDoI9Z0bTbztfNE9bG59DeyCT5DDF7OuiaK YpDFSt7DOfWgQrDfrUPqrTyXv9nTuytCZRnEsfAUoBa4nidarcY6QlBHUM+Ddn+E6pTA LPmw== X-Received: by 10.68.239.71 with SMTP id vq7mr2834903pbc.155.1415120020033; Tue, 04 Nov 2014 08:53:40 -0800 (PST) Received: from [10.0.1.8] (50-0-30-197.dsl.dynamic.fusionbroadband.com. [50.0.30.197]) by mx.google.com with ESMTPSA id ms3sm901742pdb.19.2014.11.04.08.53.37 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Nov 2014 08:53:38 -0800 (PST) Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1990.1\)) Subject: Re: ARC size limit From: George Kola In-Reply-To: Date: Tue, 4 Nov 2014 08:53:36 -0800 Message-Id: <387D6A6C-EE3E-47E7-B32C-F760C40B8635@gmail.com> References: <3B70EA0C-0976-49D6-8418-6B5D22ED7E65@gmail.com> To: Ronald Klop X-Mailer: Apple Mail (2.1990.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 16:53:40 -0000 > On Nov 4, 2014, at 1:28 AM, Ronald Klop wrote: >=20 > On Tue, 04 Nov 2014 07:29:48 +0100, George Kola > wrote: >=20 >> Hi All, >> This is my first post to freebsd-stable fresh of Meet BSD = California 2014. We are switching our entire production to FreeBSD. Our = storage servers have 256 GB of RAM , 4 TB of SSD and 40 TB of spinning = disks. We are running ZFS root and the SSD is configured as L2ARC. We = are running FreeBSD 10.1 RC3 >> I am finding that on all our machines, ARC is somehow limited = to < 64 GB of memory and we have a huge inactive memory (180 G). The = surprising thing is that ARC seems to have almost the same limit (< 64 = GB) on all of our storage boxes and ARC is not growing even though L2ARC = hit shows that there is advantage in growing ARC. >> Any help/pointers is appreciated. >> What I am trying to do is to tune ZFS for our workload. We are = hoping that we get a high hit rate. >> Thanks to Justin Gibbs and Allan Jude for initial pointers and = help. They suggested posting to the mailing list to get further help. >>=20 >> I have pasted top output and zfs-stats output below and yes UMA = is enabled. >>=20 >> Thanks, >> George >=20 > What is actually your problem? Do you just want higher numbers of ARC = usage or is your system slower than expected? > And what is your usage of these 40TB. Are all 40TB accessed a lot or = is 64 GB of ARC equal to the working set of data. >=20 > Ronald. >=20 The problem is system is slower than expected. We bumped up = the RAM from 96 GB on current illumos based production systems to 256 GB = on the new FreeBSD based system and expected higher hit rate in ARC and = lower disk access. Our rough initial calculation estimated that our = working would be between 96 - 128 GB Thanks, -George =20 =20 >=20 >=20 >>=20 >>=20 >> top >> last pid: 27458; load averages: 3.30, 5.42, 5.34 = = up 6+09:59:30 05:38:49 >> 71 processes: 1 running, 70 sleeping >> CPU: 4.2% user, 0.0% nice, 4.6% system, 0.2% interrupt, 90.9% = idle >> Mem: 11G Active, 181G Inact, 52G Wired, 1368M Cache, 4266M Free >> ARC: 47G Total, 1555M MFU, 41G MRU, 35M Anon, 3984M Header, 709M = Other >> Swap: 64G Total, 2874M Used, 61G Free, 4% Inuse >>=20 >>=20 >>=20 >> sysctl vfs.zfs.zio.use_uma >> vfs.zfs.zio.use_uma: 1 >>=20 >>=20 >>=20 >>=20 >> zfs-mon -a output >>=20 >> ZFS real-time cache activity monitor >> Seconds elapsed: 62 >>=20 >> Cache hits and misses: >> 1s 10s 60s tot >> ARC hits: 124 126 103 101 >> ARC misses: 35 46 29 28 >> ARC demand data hits: 55 90 61 61 >> ARC demand data misses: 20 32 18 17 >> ARC demand metadata hits: 69 36 42 40 >> ARC demand metadata misses: 9 13 10 9 >> ARC prefetch data hits: 0 0 0 0 >> ARC prefetch data misses: 6 1 1 1 >> ARC prefetch metadata hits: 0 0 0 0 >> ARC prefetch metadata misses: 0 0 0 0 >> L2ARC hits: 16 28 14 14 >> L2ARC misses: 19 18 15 14 >> ZFETCH hits: 592 2842 2098 2047 >> ZFETCH misses: 308 1326 507 494 >>=20 >> Cache efficiency percentage: >> 10s 60s tot >> ARC: 73.26 78.03 78.29 >> ARC demand data: 73.77 77.22 78.21 >> ARC demand metadata: 73.47 80.77 81.63 >> ARC prefetch data: 0.00 0.00 0.00 >> ARC prefetch metadata: 0.00 0.00 0.00 >> L2ARC: 60.87 48.28 50.00 >> ZFETCH: 68.19 80.54 80.56 >>=20 >>=20 >>=20 >>=20 >> zfs-stats -a output >>=20 >> ZFS real-time cache activity monitor >> Seconds elapsed: 62 >>=20 >> Cache hits and misses: >> 1s 10s 60s tot >> ARC hits: 124 126 103 101 >> ARC misses: 35 46 29 28 >> ARC demand data hits: 55 90 61 61 >> ARC demand data misses: 20 32 18 17 >> ARC demand metadata hits: 69 36 42 40 >> ARC demand metadata misses: 9 13 10 9 >> ARC prefetch data hits: 0 0 0 0 >> ARC prefetch data misses: 6 1 1 1 >> ARC prefetch metadata hits: 0 0 0 0 >> ARC prefetch metadata misses: 0 0 0 0 >> L2ARC hits: 16 28 14 14 >> L2ARC misses: 19 18 15 14 >> ZFETCH hits: 592 2842 2098 2047 >> ZFETCH misses: 308 1326 507 494 >>=20 >> Cache efficiency percentage: >> 10s 60s tot >> ARC: 73.26 78.03 78.29 >> ARC demand data: 73.77 77.22 78.21 >> ARC demand metadata: 73.47 80.77 81.63 >> ARC prefetch data: 0.00 0.00 0.00 >> ARC prefetch metadata: 0.00 0.00 0.00 >> L2ARC: 60.87 48.28 50.00 >> ZFETCH: 68.19 80.54 80.56 >>=20 >>=20 >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org = mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable = >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = " > _______________________________________________ > freebsd-stable@freebsd.org mailing = list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable = > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = "