From owner-freebsd-fs@freebsd.org Fri Aug 21 13:20:57 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2D639BD88D for ; Fri, 21 Aug 2015 13:20:57 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3BA701E34; Fri, 21 Aug 2015 13:20:57 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by lbbtg9 with SMTP id tg9so43681769lbb.1; Fri, 21 Aug 2015 06:20:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Hjp2mIxaEpprC04j6qk7G4coem6Ar2RCj7odrp9POrA=; b=BS2zVzA12uUyFz37SKRnFuryYVOSKaeDK7tlsJr07A45YACwweh+bM4blexbiJ3Nt/ M5OzVt1VgyBiadxxSTMg2VjYgQLJqUZDSzV0IfVPCDt3p3Mubo6do8bfzDVRAxjXqeh+ kZSoUpadehV3R7ScZCZmnhnkHWAQUIkGPsd+f0QqtvHSSTv//d5nOtHoGKXF0Dq1yEnE SsHKSBZumlY2jxhjd65EKPOskG7zv5v6wBMRqC03i7B/2PidVAsTnx2fJRchX0fDB5Ne NT6w62PKKkrDIJTa/hsnoTakgVlLNcGkRU4BrALeLZr0nlmD3mTywzoKfVlbTeQuZQRP hbYw== MIME-Version: 1.0 X-Received: by 10.152.28.105 with SMTP id a9mr8039256lah.9.1440163254907; Fri, 21 Aug 2015 06:20:54 -0700 (PDT) Received: by 10.112.154.33 with HTTP; Fri, 21 Aug 2015 06:20:54 -0700 (PDT) Received: by 10.112.154.33 with HTTP; Fri, 21 Aug 2015 06:20:54 -0700 (PDT) In-Reply-To: <55D71A56.2080300@FreeBSD.org> References: <0CEC2752-7787-4C6D-99E2-E7D7BF238449@omnigroup.com> <20150820002946.GD13503@in-addr.com> <55D582F9.6020207@FreeBSD.org> <55D71A56.2080300@FreeBSD.org> Date: Fri, 21 Aug 2015 16:20:54 +0300 Message-ID: Subject: Re: ZFS L2ARC statistics interpretation From: Sami Halabi To: Andriy Gapon Cc: freebsd-fs@freebsd.org, Wim Lewis , Gary Palmer Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Aug 2015 13:20:57 -0000 Will there be a patch for 10.2 ? =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 21 =D7=91=D7=90=D7=95=D7=92=D7=B3 2015= 15:33,=E2=80=8F "Andriy Gapon" =D7=9B=D7=AA=D7=91: > On 20/08/2015 10:34, Andriy Gapon wrote: > > On 20/08/2015 03:29, Gary Palmer wrote: > >> On Wed, Aug 19, 2015 at 04:08:47PM -0700, Wim Lewis wrote: > >>> I'm trying to understand some problems we've been having with our ZFS > systems, in particular their L2ARC performance. Before I make too many > guesses about what's going on, I'm hoping someone can clarify what some o= f > the ZFS statistics actually mean, or point me to documentation if any > exists. > >>> > >>> In particular, I'm hoping someone can tell me the interpretation of: > >>> > >>> Errors: > >>> kstat.zfs.misc.arcstats.l2_cksum_bad > >>> kstat.zfs.misc.arcstats.l2_io_error > >>> > >>> Other than problems with the underlying disk (or controller or cable > or...), are there reasons for these counters to be nonzero? On some of ou= r > systems, they increase fairly rapidly (20000/day). Is this considered > normal, or does it indicate a problem? If a problem, what should I be > looking at? > >>> > >>> Size: > >>> kstat.zfs.misc.arcstats.l2_size > >>> kstat.zfs.misc.arcstats.l2_asize > >>> > >>> What does l2_size/l2_asize measure? Compressed or uncompressed size? > It sometimes tops out at roughly the size of my L2ARC device, and sometim= es > just continually grows (e.g., one of my systems has an l2_size of about > 1.3T but a 190G L2ARC; I doubt I'm getting nearly 7:1 compression on my > dataset! But maybe I am? How can I tell?) > >>> > >>> There are reports over the last few years [1,2,3,4] that suggest that > there's a ZFS bug that attempts to use space past the end of the L2ARC, > resulting both in l2_size being larger than is possible and also in > io_errors and bad cksums (when the nonexistent sectors are read back). Bu= t > given that this behavior has been reported off and on for several years > now, and many of the threads devolve into supposition and folklore, I'm > hoping to get an informed answer about what these statistics mean, whethe= r > the numbers I'm seeing indicate a problem or not, and be able to make a > judgment about whether a given fix in FreeBSD might solve the problem. > >>> > >>> FWIW, I'm seeing these problems on FreeBSD 10.0 and 10.1; I'm not > seeing them on 9.2. > >>> > >>> > >>> [1] > https://lists.freebsd.org/pipermail/freebsd-current/2013-October/045088.h= tml > >>> [2] https://forums.freebsd.org/threads/l2arc-degraded.47540/ > >>> [3] > https://lists.freebsd.org/pipermail/freebsd-fs/2014-October/020256.html > >>> [4] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D198242 > >> > >> > >> I think the checksum/IO problems as well as the huge reported size > >> of your L2ARC are both a result of a problem described at the followin= g > >> url > >> > >> https://reviews.freebsd.org/D2764 > >> > >> Not sure if a fix is in 10.2 or not yet. > > > > The fix is not in head yet. > > And the patch needs to be rebased after the recent large imports of the > > upstream code. > > An updated patch for head is here > https://reviews.freebsd.org/D2764?download=3Dtrue > Testers are welcome! > > > -- > Andriy Gapon > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >