From owner-freebsd-stable@FreeBSD.ORG Tue Nov 4 09:08:11 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5078C3E3 for ; Tue, 4 Nov 2014 09:08:11 +0000 (UTC) Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D88F2D02 for ; Tue, 4 Nov 2014 09:08:09 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id q5so8106481wiv.5 for ; Tue, 04 Nov 2014 01:08:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=kK52DZT/U7LuqYu+9vDVASGU3BTtjZ9zEgM8jkvhlvQ=; b=N/8xcFfcUbkzGom7Kt1ROKE4gMF9HwEE1KGftEN8y/vPdVxWuCatgTPzIz18KULIHy 6uYsIK9017rWW58ndEsrxmbRlJCWPVVPKkaKO8ySJC+9oa4Id4cn484dXOfMXT2Kf4QY VlVst9BXOHzYnsW5bxepYcigIH3NJDPTSvMX6cG1jKTrf5n7XXTjOFedxBczePVNe1du wU8aicvGBq51FPuwrDJ3aWGNP3NVIm2O8OgovRuRLsrpMBiYD05+NeY/pITJvSRbfyk9 BtyJwHCwqbCC9zu6EasMDfrzCdHn3p9q798dPZZv/HsZgVXlDpkc6ddTREWP14W+a21O asrQ== X-Gm-Message-State: ALoCoQlHPAWpUk8ronUEiqtcKIr/zb02/agYkxFvrOdXViM2VNLrzQeMj7PNmKa+/xtUOejl3qa1 X-Received: by 10.194.200.129 with SMTP id js1mr15514671wjc.0.1415092087938; Tue, 04 Nov 2014 01:08:07 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id i3sm11552691wix.0.2014.11.04.01.08.06 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Nov 2014 01:08:06 -0800 (PST) Message-ID: <54589722.3080803@multiplay.co.uk> Date: Tue, 04 Nov 2014 09:06:42 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: ARC size limit References: <3B70EA0C-0976-49D6-8418-6B5D22ED7E65@gmail.com> In-Reply-To: <3B70EA0C-0976-49D6-8418-6B5D22ED7E65@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Nov 2014 09:08:11 -0000 You need https://svnweb.freebsd.org/base?view=revision&revision=272875 On 04/11/2014 06:29, George Kola wrote: > Hi All, > This is my first post to freebsd-stable fresh of Meet BSD California 2014. We are switching our entire production to FreeBSD. Our storage servers have 256 GB of RAM , 4 TB of SSD and 40 TB of spinning disks. We are running ZFS root and the SSD is configured as L2ARC. We are running FreeBSD 10.1 RC3 > I am finding that on all our machines, ARC is somehow limited to < 64 GB of memory and we have a huge inactive memory (180 G). The surprising thing is that ARC seems to have almost the same limit (< 64 GB) on all of our storage boxes and ARC is not growing even though L2ARC hit shows that there is advantage in growing ARC. > Any help/pointers is appreciated. > What I am trying to do is to tune ZFS for our workload. We are hoping that we get a high hit rate. > Thanks to Justin Gibbs and Allan Jude for initial pointers and help. They suggested posting to the mailing list to get further help. > > I have pasted top output and zfs-stats output below and yes UMA is enabled. > > Thanks, > George > > > > top > last pid: 27458; load averages: 3.30, 5.42, 5.34 up 6+09:59:30 05:38:49 > 71 processes: 1 running, 70 sleeping > CPU: 4.2% user, 0.0% nice, 4.6% system, 0.2% interrupt, 90.9% idle > Mem: 11G Active, 181G Inact, 52G Wired, 1368M Cache, 4266M Free > ARC: 47G Total, 1555M MFU, 41G MRU, 35M Anon, 3984M Header, 709M Other > Swap: 64G Total, 2874M Used, 61G Free, 4% Inuse > > > > sysctl vfs.zfs.zio.use_uma > vfs.zfs.zio.use_uma: 1 > > > > > zfs-mon -a output > > ZFS real-time cache activity monitor > Seconds elapsed: 62 > > Cache hits and misses: > 1s 10s 60s tot > ARC hits: 124 126 103 101 > ARC misses: 35 46 29 28 > ARC demand data hits: 55 90 61 61 > ARC demand data misses: 20 32 18 17 > ARC demand metadata hits: 69 36 42 40 > ARC demand metadata misses: 9 13 10 9 > ARC prefetch data hits: 0 0 0 0 > ARC prefetch data misses: 6 1 1 1 > ARC prefetch metadata hits: 0 0 0 0 > ARC prefetch metadata misses: 0 0 0 0 > L2ARC hits: 16 28 14 14 > L2ARC misses: 19 18 15 14 > ZFETCH hits: 592 2842 2098 2047 > ZFETCH misses: 308 1326 507 494 > > Cache efficiency percentage: > 10s 60s tot > ARC: 73.26 78.03 78.29 > ARC demand data: 73.77 77.22 78.21 > ARC demand metadata: 73.47 80.77 81.63 > ARC prefetch data: 0.00 0.00 0.00 > ARC prefetch metadata: 0.00 0.00 0.00 > L2ARC: 60.87 48.28 50.00 > ZFETCH: 68.19 80.54 80.56 > > > > > zfs-stats -a output > > ZFS real-time cache activity monitor > Seconds elapsed: 62 > > Cache hits and misses: > 1s 10s 60s tot > ARC hits: 124 126 103 101 > ARC misses: 35 46 29 28 > ARC demand data hits: 55 90 61 61 > ARC demand data misses: 20 32 18 17 > ARC demand metadata hits: 69 36 42 40 > ARC demand metadata misses: 9 13 10 9 > ARC prefetch data hits: 0 0 0 0 > ARC prefetch data misses: 6 1 1 1 > ARC prefetch metadata hits: 0 0 0 0 > ARC prefetch metadata misses: 0 0 0 0 > L2ARC hits: 16 28 14 14 > L2ARC misses: 19 18 15 14 > ZFETCH hits: 592 2842 2098 2047 > ZFETCH misses: 308 1326 507 494 > > Cache efficiency percentage: > 10s 60s tot > ARC: 73.26 78.03 78.29 > ARC demand data: 73.77 77.22 78.21 > ARC demand metadata: 73.47 80.77 81.63 > ARC prefetch data: 0.00 0.00 0.00 > ARC prefetch metadata: 0.00 0.00 0.00 > L2ARC: 60.87 48.28 50.00 > ZFETCH: 68.19 80.54 80.56 > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"