From owner-freebsd-fs@FreeBSD.ORG Fri Feb 24 12:51:16 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C1A9F1065670 for ; Fri, 24 Feb 2012 12:51:16 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 724B38FC16 for ; Fri, 24 Feb 2012 12:51:16 +0000 (UTC) Received: by vcge1 with SMTP id e1so64238vcg.13 for ; Fri, 24 Feb 2012 04:51:15 -0800 (PST) Received-SPF: pass (google.com: domain of tevans.uk@googlemail.com designates 10.52.27.99 as permitted sender) client-ip=10.52.27.99; Authentication-Results: mr.google.com; spf=pass (google.com: domain of tevans.uk@googlemail.com designates 10.52.27.99 as permitted sender) smtp.mail=tevans.uk@googlemail.com; dkim=pass header.i=tevans.uk@googlemail.com Received: from mr.google.com ([10.52.27.99]) by 10.52.27.99 with SMTP id s3mr1024903vdg.121.1330087875946 (num_hops = 1); Fri, 24 Feb 2012 04:51:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=2NW5fOcjjAb+cWWaIW+p+keCsFgDdnBAusvA+yIidiE=; b=DkROjfiz3did87s/fCHlXls2Z5hXV9+MIKVW77yaoHPtvTyuA9RJYQv3DU/AAfG2IZ aT3+XgZ3KFaFqlif4LhW+R+wNBHFsXETY1r4j4RRKIRGiuEjgr8tjP4LN27q3y9lPDxq MhFcvUQQlJ05ld33c0frrG8ySl/FT6IQSiEb8= MIME-Version: 1.0 Received: by 10.52.27.99 with SMTP id s3mr766254vdg.121.1330086098209; Fri, 24 Feb 2012 04:21:38 -0800 (PST) Received: by 10.52.91.210 with HTTP; Fri, 24 Feb 2012 04:21:38 -0800 (PST) In-Reply-To: <1330081612.13430.39.camel@pow> References: <1330081612.13430.39.camel@pow> Date: Fri, 24 Feb 2012 12:21:38 +0000 Message-ID: From: Tom Evans To: Luke Marsden Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, team@hybrid-logic.co.uk Subject: Re: Another ZFS ARC memory question X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Feb 2012 12:51:16 -0000 On Fri, Feb 24, 2012 at 11:06 AM, Luke Marsden wrote: > Hi all, > > Just wanted to get your opinion on best practices for ZFS. > > We're running 8.2-RELEASE v15 in production on 24GB RAM amd64 machines > but have been having trouble with short spikes in application memory > usage resulting in huge amounts of swapping, bringing the whole machine > to its knees and crashing it hard. =C2=A0I suspect this is because when t= here > is a sudden spike in memory usage the zfs arc reclaim thread is unable > to free system memory fast enough. > > This most recently happened yesterday as you can see from the following > munin graphs: > > E.g. http://hybrid-logic.co.uk/memory-day.png > =C2=A0 =C2=A0 http://hybrid-logic.co.uk/swap-day.png > > Our response has been to start limiting the ZFS ARC cache to 4GB on our > production machines - trading performance for stability is fine with me > (and we have L2ARC on SSD so we still get good levels of caching). > > My questions are: > > =C2=A0 =C2=A0 =C2=A0* is this a known problem? > =C2=A0 =C2=A0 =C2=A0* what is the community's advice for production machi= nes running > =C2=A0 =C2=A0 =C2=A0 =C2=A0ZFS on FreeBSD, is manually limiting the ARC c= ache (to ensure > =C2=A0 =C2=A0 =C2=A0 =C2=A0that there's enough actually free memory to ha= ndle a spike in > =C2=A0 =C2=A0 =C2=A0 =C2=A0application memory usage) the best solution to= this > =C2=A0 =C2=A0 =C2=A0 =C2=A0spike-in-memory-means-crash problem? > =C2=A0 =C2=A0 =C2=A0* has FreeBSD 9.0 / ZFS v28 solved this problem? > =C2=A0 =C2=A0 =C2=A0* rather than setting a hard limit on the ARC cache s= ize, is it > =C2=A0 =C2=A0 =C2=A0 =C2=A0possible to adjust the auto-tuning variables t= o leave more free > =C2=A0 =C2=A0 =C2=A0 =C2=A0memory for spiky memory situations? =C2=A0e.g.= set the auto-tuning to > =C2=A0 =C2=A0 =C2=A0 =C2=A0make arc eat 80% of memory instead of ~95% lik= e it is at > =C2=A0 =C2=A0 =C2=A0 =C2=A0present? > =C2=A0 =C2=A0 =C2=A0* could the arc reclaim thread be made to drop ARC pa= ges with > =C2=A0 =C2=A0 =C2=A0 =C2=A0higher priority before the system starts swapp= ing out > =C2=A0 =C2=A0 =C2=A0 =C2=A0application pages? > > Thank you for any/all answers, and thank you for making FreeBSD > awesome :-) It's not a problem, it's a feature! By default the ARC will attempt to cache as much as it can - it assumes the box is a ZFS filer, and doesn't need RAM for applications. The solution, as you've found out, is to limit how much ARC can take up. In practice, you should be doing this anyway. You should know, or have an idea, of how much RAM is required for the applications on that box, and you need to limit ZFS to not eat into that required RAM. Cheers Tom