From owner-freebsd-fs@freebsd.org Fri Aug 21 13:28:33 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 780AA9BD911 for ; Fri, 21 Aug 2015 13:28:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 8663767; Fri, 21 Aug 2015 13:28:32 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id QAA25292; Fri, 21 Aug 2015 16:28:30 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1ZSmMv-000Drh-Py; Fri, 21 Aug 2015 16:28:30 +0300 Subject: Re: ZFS L2ARC statistics interpretation To: Sami Halabi References: <0CEC2752-7787-4C6D-99E2-E7D7BF238449@omnigroup.com> <20150820002946.GD13503@in-addr.com> <55D582F9.6020207@FreeBSD.org> <55D71A56.2080300@FreeBSD.org> Cc: freebsd-fs@FreeBSD.org, Wim Lewis , Gary Palmer From: Andriy Gapon Message-ID: <55D7272F.6020106@FreeBSD.org> Date: Fri, 21 Aug 2015 16:27:11 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Aug 2015 13:28:33 -0000 On 21/08/2015 16:20, Sami Halabi wrote: > Will there be a patch for 10.2 ? There was a patch against older version of head that should have been applicable to 10.2. It still can be accessed via https://reviews.freebsd.org/D2764?vs=on&id=6055&whitespace=ignore-most&download=true > בתאריך 21 באוג׳ 2015 15:33,‏ "Andriy Gapon" > כתב: > > On 20/08/2015 10:34, Andriy Gapon wrote: > > On 20/08/2015 03:29, Gary Palmer wrote: > >> On Wed, Aug 19, 2015 at 04:08:47PM -0700, Wim Lewis wrote: > >>> I'm trying to understand some problems we've been having with > our ZFS systems, in particular their L2ARC performance. Before I > make too many guesses about what's going on, I'm hoping someone can > clarify what some of the ZFS statistics actually mean, or point me > to documentation if any exists. > >>> > >>> In particular, I'm hoping someone can tell me the interpretation of: > >>> > >>> Errors: > >>> kstat.zfs.misc.arcstats.l2_cksum_bad > >>> kstat.zfs.misc.arcstats.l2_io_error > >>> > >>> Other than problems with the underlying disk (or controller or > cable or...), are there reasons for these counters to be nonzero? On > some of our systems, they increase fairly rapidly (20000/day). Is > this considered normal, or does it indicate a problem? If a problem, > what should I be looking at? > >>> > >>> Size: > >>> kstat.zfs.misc.arcstats.l2_size > >>> kstat.zfs.misc.arcstats.l2_asize > >>> > >>> What does l2_size/l2_asize measure? Compressed or uncompressed > size? It sometimes tops out at roughly the size of my L2ARC device, > and sometimes just continually grows (e.g., one of my systems has an > l2_size of about 1.3T but a 190G L2ARC; I doubt I'm getting nearly > 7:1 compression on my dataset! But maybe I am? How can I tell?) > >>> > >>> There are reports over the last few years [1,2,3,4] that suggest > that there's a ZFS bug that attempts to use space past the end of > the L2ARC, resulting both in l2_size being larger than is possible > and also in io_errors and bad cksums (when the nonexistent sectors > are read back). But given that this behavior has been reported off > and on for several years now, and many of the threads devolve into > supposition and folklore, I'm hoping to get an informed answer about > what these statistics mean, whether the numbers I'm seeing indicate > a problem or not, and be able to make a judgment about whether a > given fix in FreeBSD might solve the problem. > >>> > >>> FWIW, I'm seeing these problems on FreeBSD 10.0 and 10.1; I'm > not seeing them on 9.2. > >>> > >>> > >>> [1] > https://lists.freebsd.org/pipermail/freebsd-current/2013-October/045088.html > >>> [2] https://forums.freebsd.org/threads/l2arc-degraded.47540/ > >>> [3] > https://lists.freebsd.org/pipermail/freebsd-fs/2014-October/020256.html > >>> [4] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198242 > >> > >> > >> I think the checksum/IO problems as well as the huge reported size > >> of your L2ARC are both a result of a problem described at the > following > >> url > >> > >> https://reviews.freebsd.org/D2764 > >> > >> Not sure if a fix is in 10.2 or not yet. > > > > The fix is not in head yet. > > And the patch needs to be rebased after the recent large imports > of the > > upstream code. > > An updated patch for head is here > https://reviews.freebsd.org/D2764?download=true > Testers are welcome! > > > -- > Andriy Gapon > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org > " > -- Andriy Gapon