From owner-freebsd-current@FreeBSD.ORG Thu Oct 10 10:01:15 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2565DF64 for ; Thu, 10 Oct 2013 10:01:15 +0000 (UTC) (envelope-from satan@ukr.net) Received: from hell.ukr.net (hell.ukr.net [212.42.67.68]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D597F2E9A for ; Thu, 10 Oct 2013 10:01:14 +0000 (UTC) Received: from satan by hell.ukr.net with local ID 1VUD3Q-000F6O-SF ; Thu, 10 Oct 2013 13:01:12 +0300 Date: Thu, 10 Oct 2013 13:01:12 +0300 From: Vitalij Satanivskij To: Vitalij Satanivskij Subject: Re: ZFS L2ARC - incorrect size and abnormal system load on r255173 Message-ID: <20131010100112.GA57736@hell.ukr.net> References: <1381166916.122992963.5h9ygiri@frv45.ukr.net> <1381170764.32684.31088349.343931EE@webmail.messagingengine.com> <20131007185032.GA82932@hell.ukr.net> <20131007211201.GA89306@hell.ukr.net> <20131010092223.GA28347@hell.ukr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131010092223.GA28347@hell.ukr.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Oct 2013 10:01:15 -0000 Hm, another strange thing's on another server - zfs-stats -L ------------------------------------------------------------------------ ZFS Subsystem Report Thu Oct 10 12:56:54 2013 ------------------------------------------------------------------------ L2 ARC Summary: (DEGRADED) Passed Headroom: 8.34m Tried Lock Failures: 145.66m IO In Progress: 9.76m Low Memory Aborts: 526 Free on Write: 1.70m Writes While Full: 29.28k R/W Clashes: 341.30k Bad Checksums: 865.91k IO Errors: 44.19k SPA Mismatch: 32.03m L2 ARC Size: (Adaptive) 189.28 GiB Header Size: 4.88% 9.24 GiB Looks like size have nothing similar with IO errors. So question is - when error's like Bad Checksums and IO Errors can hapend? Look's like no hardware problem's. All ssd atached to onboard intel sata controler (Motherboard is Supermicro X9SRL-F) Vitalij Satanivskij wrote: VS> Same situation hapend yesterday again :( VS> VS> What's confuse me while trying to understend where I'm wrong VS> VS> VS> Firt some info. VS> VS> We have zfs pool "POOL" and one more zfs on it "POOL/zfs" VS> VS> POOL - have only primarycache enabled "ALL" VS> POOL/zfs - have both primay and secondary for "ALL" VS> VS> POOL have compression=lz4 VS> VS> POOL/zfs have none VS> VS> VS> POOL - have around 9TB data VS> VS> POOL/zfs - have 1TB VS> VS> Secondary cache have configuration - VS> VS> cache VS> gpt/cache0 ONLINE 0 0 0 VS> gpt/cache1 ONLINE 0 0 0 VS> gpt/cache2 ONLINE 0 0 0 VS> VS> gpt/cache0-2 it's intel sdd SSDSC2BW180A4 180gb VS> VS> So full real size for l2 is 540GB (realy 489gb) VS> VS> First question - data on l2arc will be compressed on not? VS> VS> Second in stats we see VS> VS> L2 ARC Size: (Adaptive) 2.08 TiB VS> VS> eary it was 1.1 1.4 ... VS> VS> So a) how cache can be biger than zfs it self VS> b) in case it's not compressed (answer for first question) how it an be biger than real ssd size? VS> VS> VS> one more coment if l2 arc size grove above phisical sizes I se next stats VS> VS> kstat.zfs.misc.arcstats.l2_cksum_bad: 50907344 VS> kstat.zfs.misc.arcstats.l2_io_error: 4547377 VS> VS> and growing. VS> VS> VS> System is r255173 with patch from rr255173 VS> VS> VS> At last maybe somebody have any ideas what's realy hapend... VS> VS> VS> VS> VS> VS> Vitalij Satanivskij wrote: VS> VS> VS> VS> One more question - VS> VS> VS> VS> we have two counter - VS> VS> VS> VS> kstat.zfs.misc.arcstats.l2_size: 1256609410560 VS> VS> kstat.zfs.misc.arcstats.l2_asize: 1149007667712 VS> VS> VS> VS> can anybody explain how to understand them i.e. l2_asize - real used space on l2arc an l2_size - uncompressed size, VS> VS> VS> VS> or maybe something else ? VS> VS> VS> VS> VS> VS> VS> VS> Vitalij Satanivskij wrote: VS> VS> VS> VS> VS> VS> Data on pool have compressratio around 1.4 VS> VS> VS> VS> VS> VS> On diferent servers with same data type and load L2 ARC Size: (Adaptive) can be diferent VS> VS> VS> VS> VS> VS> for example 1.04 TiB vs 1.45 TiB VS> VS> VS> VS> VS> VS> But it's all have same porblem - grow in time. VS> VS> VS> VS> VS> VS> VS> VS> VS> More stange for us - VS> VS> VS> VS> VS> VS> ARC: 80G Total, 4412M MFU, 5040M MRU, 76M Anon, 78G Header, 2195M Other VS> VS> VS> VS> VS> VS> 78G header size and ubnormal - VS> VS> VS> VS> VS> VS> kstat.zfs.misc.arcstats.l2_cksum_bad: 210920592 VS> VS> VS> kstat.zfs.misc.arcstats.l2_io_error: 7362414 VS> VS> VS> VS> VS> VS> sysctl's growing avery second. VS> VS> VS> VS> VS> VS> All part's of server (as hardware part's) in in normal state. VS> VS> VS> VS> VS> VS> After reboot no problem's for some period untile cache size grow to some limit. VS> VS> VS> VS> VS> VS> VS> VS> VS> VS> VS> VS> Mark Felder wrote: VS> VS> VS> MF> On Mon, Oct 7, 2013, at 13:09, Dmitriy Makarov wrote: VS> VS> VS> MF> > VS> VS> VS> MF> > How can L2 ARC Size: (Adaptive) be 1.44 TiB (up) with total physical size VS> VS> VS> MF> > of L2ARC devices 490GB? VS> VS> VS> MF> > VS> VS> VS> MF> VS> VS> VS> MF> http://svnweb.freebsd.org/base?view=revision&revision=251478 VS> VS> VS> MF> VS> VS> VS> MF> L2ARC compression perhaps? VS> VS> VS> MF> _______________________________________________ VS> VS> VS> MF> freebsd-current@freebsd.org mailing list VS> VS> VS> MF> http://lists.freebsd.org/mailman/listinfo/freebsd-current VS> VS> VS> MF> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" VS> VS> VS> _______________________________________________ VS> VS> VS> freebsd-current@freebsd.org mailing list VS> VS> VS> http://lists.freebsd.org/mailman/listinfo/freebsd-current VS> VS> VS> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" VS> VS> _______________________________________________ VS> VS> freebsd-current@freebsd.org mailing list VS> VS> http://lists.freebsd.org/mailman/listinfo/freebsd-current VS> VS> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" VS> _______________________________________________ VS> freebsd-current@freebsd.org mailing list VS> http://lists.freebsd.org/mailman/listinfo/freebsd-current VS> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"