From owner-freebsd-bugs@FreeBSD.ORG Fri Feb 13 16:04:45 2015 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1EA9171A for ; Fri, 13 Feb 2015 16:04:45 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 005B31F7 for ; Fri, 13 Feb 2015 16:04:45 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t1DG4iK2072832 for ; Fri, 13 Feb 2015 16:04:44 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 195746] zfs L2ARC wrong alloc/free size Date: Fri, 13 Feb 2015 16:04:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: john@infoassurancepro.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Feb 2015 16:04:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195746 John K. Gates changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |john@infoassurancepro.com --- Comment #1 from John K. Gates --- I have been discussing this issue here: https://forums.freebsd.org/threads/l2arc-degraded.47540/ Checksum and IO errors appear after an L2ARC device fills completely with cache data on any release of FreeBSD after L2ARC compression (http://wiki.illumos.org/display/illumos/L2ARC+Compression) was enabled (9.3-RELEASE and later, and 10.0-RELEASE and later). The following is from an 10.1-RELEASE-p5 (svn rev 278678) running a generic kernel. /etc/make.conf contains only "CPUTYPE?=core2". ZFS pools are v28 (created under 9.2-RELEASE) and have NOT been updated for feature flags. "zfs-stats -L" shows the L2ARC "DEGRADED" with numerous I/O and checksum errors: root@cadence:/ # zfs-stats -L ------------------------------------------------------------------------ ZFS Subsystem Report Fri Feb 13 10:36:49 2015 ------------------------------------------------------------------------ L2 ARC Summary: (DEGRADED) Passed Headroom: 30.28m Tried Lock Failures: 24.83m IO In Progress: 247 Low Memory Aborts: 103 Free on Write: 54.97k Writes While Full: 10.62k R/W Clashes: 562 Bad Checksums: 1.29m IO Errors: 128.28k SPA Mismatch: 48.53b L2 ARC Size: (Adaptive) 33.51 GiB Header Size: 2.24% 768.85 MiB L2 ARC Evicts: Lock Retries: 18 Upon Reading: 0 L2 ARC Breakdown: 35.47m Hit Ratio: 26.64% 9.45m Miss Ratio: 73.36% 26.02m Feeds: 568.79k L2 ARC Buffer: Bytes Scanned: 530.43 TiB Buffer Iterations: 568.79k List Iterations: 36.17m NULL List Iterations: 974.71k L2 ARC Writes: Writes Sent: 100.00% 136.10k ------------------------------------------------------------------------ Kernel variables showing compression is working on L2ARC, and there are I/O and checksum errors: root@cadence:/ # sysctl kstat.zfs.misc.arcstats.l2_compress_successes kstat.zfs.misc.arcstats.l2_compress_zeros kstat.zfs.misc.arcstats.l2_compress_failures kstat.zfs.misc.arcstats.l2_cksum_bad kstat.zfs.misc.arcstats.l2_io_error kstat.zfs.misc.arcstats.l2_compress_successes: 1353514 kstat.zfs.misc.arcstats.l2_compress_zeros: 29 kstat.zfs.misc.arcstats.l2_compress_failures: 4985 kstat.zfs.misc.arcstats.l2_cksum_bad: 1290021 kstat.zfs.misc.arcstats.l2_io_error: 128275 Slight variations of this problem have been reported in numerous instances on FreeBSD, FreeNAS and PC-BSD related forums and mailing lists, and is usually dismissed as a hardware problem: https://bugs.freenas.org/issues/5347 https://forums.freenas.org/index.php?threads/l2-arc-summary-degraded.19256/ https://bugs.pcbsd.org/issues/3418 http://svnweb.freebsd.org/base?view=revision&sortby=file&revision=256889 http://lists.freebsd.org/pipermail/freebsd-current/2013-October/045088.html http://lists.freebsd.org/pipermail/freebsd-bugs/2014-November/059261.html http://lists.freebsd.org/pipermail/freebsd-bugs/2014-December/059376.html http://lists.freebsd.org/pipermail/freebsd-fs/2014-October/020242.html However, I've been able to easily duplicate this problem on two different sets of high quality, reliable hardware (Dell PowerEdge, Intel SSD) that otherwise tests perfectly. To duplicate, simply create a zfs pool with a small L2ARC device and exercise the pool with random I/O until the L2ARC fills. -- You are receiving this mail because: You are the assignee for the bug.