From owner-freebsd-bugs@FreeBSD.ORG Tue Mar 3 16:15:44 2015 Return-Path: Delivered-To: freebsd-bugs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2A5A3AE2 for ; Tue, 3 Mar 2015 16:15:44 +0000 (UTC) Received: from smtp.bredband2.com (smtp.bredband2.com [83.219.192.166]) by mx1.freebsd.org (Postfix) with ESMTP id AF78410A for ; Tue, 3 Mar 2015 16:15:43 +0000 (UTC) Received: from tornet10.dyndns.org (c-83-233-142-148.cust.bredband2.com [83.233.142.148]) (Authenticated sender: ed5470) by smtp.bredband2.com (Postfix) with ESMTPA id 53A50E4805 for ; Tue, 3 Mar 2015 17:06:33 +0100 (CET) Received: from ASSP.nospam (localhost [127.0.0.1]) by tornet10.dyndns.org (Postfix) with ESMTP id 180DF1CC5F for ; Tue, 3 Mar 2015 17:06:33 +0100 (CET) Received: from anyhost.local ([127.0.0.1] helo=anyhost.local) by ASSP.nospam with ESMTP (ASSP 1.9.9); 3 Mar 2015 17:06:32 +0100 Message-ID: <54F5DC07.90200@tornet10.dyndns.org> Date: Tue, 03 Mar 2015 17:06:31 +0100 From: kenneth User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: freebsd-bugs@freebsd.org Subject: L2ARC problems Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Mar 2015 16:15:44 -0000 When the L2ARC has filled up and is in use for a while problems like I/O errors and Bad checksums. The size of the L2ARC is only 80G. It looks like compression is used. # uname -v FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC # zfs-stats -L ------------------------------------------------------------------------ ZFS Subsystem Report Tue Mar 3 16:54:36 2015 ------------------------------------------------------------------------ L2 ARC Summary: (DEGRADED) Passed Headroom: 59.38m Tried Lock Failures: 7.80m IO In Progress: 14.35k Low Memory Aborts: 21 Free on Write: 27.55k Writes While Full: 8.28k R/W Clashes: 108 Bad Checksums: 14.48k IO Errors: 12.88k SPA Mismatch: 990.88k L2 ARC Size: (Adaptive) 156.25 GiB Header Size: 0.25% 405.65 MiB L2 ARC Evicts: Lock Retries: 145 Upon Reading: 0 L2 ARC Breakdown: 9.14m Hit Ratio: 11.15% 1.02m Miss Ratio: 88.85% 8.12m Feeds: 2.53m L2 ARC Buffer: Bytes Scanned: 6.50 PiB Buffer Iterations: 2.53m List Iterations: 161.70m NULL List Iterations: 340.27k L2 ARC Writes: Writes Sent: 100.00% 90.11k # sysctl -a | grep l2 vfs.zfs.l2arc_write_max: 33554432 vfs.zfs.l2arc_write_boost: 67108864 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_norw: 1 vfs.zfs.l2c_only_size: 155331604480 vfs.cache.numfullpathfail2: 0 kstat.zfs.misc.arcstats.evict_l2_cached: 684731582464 kstat.zfs.misc.arcstats.evict_l2_eligible: 98364371968 kstat.zfs.misc.arcstats.evict_l2_ineligible: 31332678656 kstat.zfs.misc.arcstats.l2_hits: 1019237 kstat.zfs.misc.arcstats.l2_misses: 8123270 kstat.zfs.misc.arcstats.l2_feeds: 2527815 kstat.zfs.misc.arcstats.l2_rw_clash: 108 kstat.zfs.misc.arcstats.l2_read_bytes: 15603715072 kstat.zfs.misc.arcstats.l2_write_bytes: 351871377408 kstat.zfs.misc.arcstats.l2_writes_sent: 90108 kstat.zfs.misc.arcstats.l2_writes_done: 90108 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 321 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 145 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 27554 kstat.zfs.misc.arcstats.l2_abort_lowmem: 21 kstat.zfs.misc.arcstats.l2_cksum_bad: 14482 kstat.zfs.misc.arcstats.l2_io_error: 12877 kstat.zfs.misc.arcstats.l2_size: 167769063936 kstat.zfs.misc.arcstats.l2_asize: 163325760512 kstat.zfs.misc.arcstats.l2_hdr_size: 425358688 kstat.zfs.misc.arcstats.l2_compress_successes: 1048332 kstat.zfs.misc.arcstats.l2_compress_zeros: 0 kstat.zfs.misc.arcstats.l2_compress_failures: 876054 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 7795393 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 59396052 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 990877 kstat.zfs.misc.arcstats.l2_write_in_l2: 142082046580 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 14352 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 116138067597 kstat.zfs.misc.arcstats.l2_write_full: 8276 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 2527815 kstat.zfs.misc.arcstats.l2_write_pios: 90108 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 7317667389325824 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 161682755 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 340271