From owner-freebsd-bugs@FreeBSD.ORG Sat Nov 29 12:57:48 2014 Return-Path: Delivered-To: freebsd-bugs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF66CA05 for ; Sat, 29 Nov 2014 12:57:48 +0000 (UTC) Received: from relay1.speechpro.com (relay1.speechpro.com [79.134.214.22]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CABEF3B4 for ; Sat, 29 Nov 2014 12:57:47 +0000 (UTC) Received: from mail.stc ([192.168.8.64] helo=spb2.speechpro.com) by relay1.speechpro.com with esmtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1XuhIB-0002iX-UZ for freebsd-bugs@freebsd.org; Sat, 29 Nov 2014 15:38:27 +0300 Received: from [79.134.214.12] (helo=mail.speechpro.com) by mail.stc with esmtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1XuhIB-0003mR-S6 for freebsd-bugs@freebsd.org; Sat, 29 Nov 2014 15:38:27 +0300 Received: from [95.140.92.4] (helo=[127.0.0.1]) by mail.speechpro.com with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1XuhIB-0000th-OQ for freebsd-bugs@freebsd.org; Sat, 29 Nov 2014 15:38:27 +0300 Message-ID: <5479BE37.6070904@speechpro.com> Date: Sat, 29 Nov 2014 15:38:15 +0300 From: =?UTF-8?B?0KLQsNCx0L7Qu9C40L0g0K7RgNC40Lk=?= User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-bugs@freebsd.org Subject: zfs L2ARC wrong alloc/free size and DEGRADED status X-Archived: Yes Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Nov 2014 12:57:49 -0000 Hi All. I have server FreeBSD 10.1-RELEASE r274401 GENERIC kernel amd64 with 2 zfs pools. There are two Intel 480Gb SSD disks in server, used like ZIL (mirror 4GB per pool) and L2ARC (stripe 75+75 GB per pool). Some days ago I noticed wrong L2ARC size pool1 in zpool iostat -v, later I seen same wrong size in pool2. Now it looks like: poolallocfreereadwritereadwrite ------------------------------------------------------------------ pool113,0T34,3T453,56K3,93M51,9M raidz313,0T34,3T453,51K3,93M47,1M multipath/pd01--3197311K4,96M multipath/pd02--3197311K4,96M multipath/pd03--3197311K4,96M multipath/pd04--3197311K4,96M multipath/pd05--3197311K4,96M multipath/pd06--3197311K4,96M multipath/pd07--3197311K4,96M multipath/pd08--3197311K4,96M multipath/pd09--3197311K4,96M multipath/pd10--3197311K4,96M multipath/pd11--3197311K4,96M multipath/pd12--3197311K4,96M multipath/pd13--3197311K4,96M logs------ mirror812K3,97G04504,83M diskid/DISK-CVWL435200Y1480QGNp1--04544,83M diskid/DISK-CVWL4353000F480QGNp1--04544,83M cache------ diskid/DISK-CVWL435200Y1480QGNp4371G16,0E427163K3,16M diskid/DISK-CVWL4353000F480QGNp4441G16,0E825145K2,94M ------------------------------------------------------------------ pool210,2T37,0T811,36K9,82M80,2M raidz310,2T37,0T818709,82M45,9M multipath/pd14--2182903K4,67M multipath/pd15--2182903K4,67M multipath/pd16--2182903K4,67M multipath/pd17--2182903K4,67M multipath/pd18--2182904K4,67M multipath/pd19--2182903K4,67M multipath/pd20--2182903K4,67M multipath/pd21--2182903K4,67M multipath/pd22--2182903K4,67M multipath/pd23--2182903K4,67M multipath/pd24--2182903K4,67M multipath/pd25--2182903K4,67M multipath/pd26--2182903K4,67M logs------ mirror238M3,74G0525034,3M diskid/DISK-CVWL435200Y1480QGNp2--0525434,3M diskid/DISK-CVWL4353000F480QGNp2--0525434,3M cache------ diskid/DISK-CVWL435200Y1480QGNp5207G16,0E12145,1K2,56M diskid/DISK-CVWL4353000F480QGNp5203G16,0E22194,6K2,60M Values “371G16,0E” are abnormal, real aloc size is 75G. After that I looked zfs-stat -L and seen DEGRADED L2ARC and too big L2ARC size: L2 ARC Summary: (DEGRADED) Passed Headroom:6.05m Tried Lock Failures:22.36m IO In Progress:2.75k Low Memory Aborts:2.86k Free on Write:5.48m Writes While Full:339.48k R/W Clashes:2.07k Bad Checksums:211.52k IO Errors:101.41k SPA Mismatch:3.16b L2 ARC Size: (Adaptive)1.27TiB Header Size:1.42%18.56GiB kstat.zfs.misc.arcstats.l2_io_error: 101531 kstat.zfs.misc.arcstats.l2_cksum_bad: 211782 smartctl told that both SSD is fine, without any IO errors. I found same issues which described about L2ARC compression: http://forums.freebsd.org/threads/l2arc-degraded.47540/ http://lists.freebsd.org/pipermail/freebsd-current/2013-October/045088.html My problem looks like same bug. Thanks for any help! -- Best regards, Tabolin Yuriy System administrator Speech Technology Center