From owner-freebsd-fs@freebsd.org Mon Aug 3 02:27:20 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 174339B267E for ; Mon, 3 Aug 2015 02:27:20 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: from mail-qg0-f44.google.com (mail-qg0-f44.google.com [209.85.192.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D5D961989 for ; Mon, 3 Aug 2015 02:27:19 +0000 (UTC) (envelope-from paul@kraus-haus.org) Received: by qgeu79 with SMTP id u79so80540004qge.1 for ; Sun, 02 Aug 2015 19:27:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:content-transfer-encoding:message-id:references:to; bh=zR2E4ChgESQCRHIJ1gHsWdFW4j686tE5jifccmnJFl8=; b=MBi6/4BGwqmLFKO9QcGMh8BiH9WzJqVemHyqZDkrQPINXhk5ewu3di8uYlnid56A64 5E3/FXvjsgq7wYythc7TZdtzn9L40GngH3PzdO+rpwN7nPU3m1iCuzeRdt9eHUW4YC6K 6ODS+TTollvcY6eLwqNBXKSJR/jlGZHg0RXKgpgxcG0k/5s2A8Ay9NRMV8ku2D4d+0UN Dd2kuA1ZVqDUm79qx45avcKYxycIR89/pZMl0iQi+dyNQiLs46jqkZx2h0Z8A5yu42ns SxsjdlGB+u6gVbOCbCx8y8vPcnvNjayqOaNvdHMmtuAroeU5EUKF+qwWC5OqOfEEvwmd TdcA== X-Gm-Message-State: ALoCoQncPF8qiwajGhwPE0TAfVFLRHEKQpolmOPgG4SjJ4y5bUFAzgLLq4uHq+IjtvHgpZSWDPtS X-Received: by 10.140.236.22 with SMTP id h22mr5414468qhc.20.1438567522589; Sun, 02 Aug 2015 19:05:22 -0700 (PDT) Received: from [192.168.2.137] (pool-100-4-179-8.albyny.fios.verizon.net. [100.4.179.8]) by smtp.gmail.com with ESMTPSA id h49sm6186357qgd.24.2015.08.02.19.05.20 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 02 Aug 2015 19:05:20 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: ZFS: Disabling ARC? From: Paul Kraus In-Reply-To: <55BC14B7.9010009@sneakertech.com> Date: Sun, 2 Aug 2015 22:05:18 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <9DBE58C6-8C42-498B-AB66-7D9BBDFAA90F@kraus-haus.org> References: <55BC14B7.9010009@sneakertech.com> To: FreeBSD FS X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 02:27:20 -0000 On Jul 31, 2015, at 20:37, Quartz wrote: > Can someone help clear up a few ZFS basics for me? >=20 > A few recent threads about ARC issues and memory-induced panics have = made me realize I'm not 100% sure I understand ARC as well as I thought = I did. > If I understand ARC correctly this would be a worst case scenario, = right? Besides hogging ram, would ARC cause any problems here? Would = disabling ARC and devoting the ram to other things be a wise idea? Is = disabling ARC ever a wise idea? The ZFS ARC is both powerful and often misunderstood. In order to do = anything intelligently with the ARC (and it=92s tunings) you need to = first know what it is doing=85 get a copy of arcstat.pl from here = https://github.com/mharsch/arcstat/blob/master/arcstat.pl =85 I prefer = it to the zfs-stats port, but any tool that shows us the realtime ARC = usage is what you need. The ARC is _supposed_ to use whatever RAM is otherwise unused _and_ if = other processes need RAM, the ARC is supposed to give it back. I know = there have been a number of issues with the ARC not freeing RAM fast = enough, but I have not seen that issue under 10.1 (I have seen it under = 9.x). On my main 9.x VM host I leave an 'arcstat.pl 60' running all the = time so I can quickly see if it is freeing as it should. Also note that the ARC does not cache files but blocks (if, as Alex = mentioned) it is cacheing data at all (or just metadata). If you are really worried about the ARC hogging RAM, then set a cap. The = kernel tunables here are: [ppk@FreeBSD2 ~]$ sysctl -a | grep zfs | grep arc vfs.zfs.arc_max: 12884901888 vfs.zfs.arc_min: 1919138304 vfs.zfs.arc_meta_used: 3221144040 vfs.zfs.arc_meta_limit: 3221225472 vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_norw: 1 =85 followed by _lots_ of arcstats, which tell you what the ARC is = doing.=20 The first one is what you really want to tune if you are RAM limited. It = sets a hard cap on how much RAM the system will use for the entire ARC. = On my home server I have the following tunings in /boot/loader.conf [ppk@FreeBSD2 /boot]$ cat loader.conf=20 zfs_load=3D"YES" # set ARC max to 12 GB with 16 GB of RAM vfs.zfs.arc_max=3D"12884901888" So I set the ARC maximum to 12 GB to leave me 4 GB for OS and = applications. I have production servers with between 24 and 80 GB of RAM and do not = generally limit the ARC but let it =93float=94 and give and take as RAM = is available. Please note that I have never seen a _panic_ due to ARC RAM issues, I = have had systems starved for RAM for periods and processes (VMs) get = very angry, but the system as a whole usually recovers. I then restart = the processes that got angry.=20 -- Paul Kraus paul@kraus-haus.org