From owner-freebsd-current@FreeBSD.ORG Fri Oct 25 06:11:58 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BDFD541F for ; Fri, 25 Oct 2013 06:11:58 +0000 (UTC) (envelope-from satan@ukr.net) Received: from hell.ukr.net (hell.ukr.net [212.42.67.68]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 708B12E75 for ; Fri, 25 Oct 2013 06:11:57 +0000 (UTC) Received: from satan by hell.ukr.net with local ID 1VZack-00018i-AI ; Fri, 25 Oct 2013 09:11:54 +0300 Date: Fri, 25 Oct 2013 09:11:54 +0300 From: Vitalij Satanivskij To: Vitalij Satanivskij Subject: Re: ZFS L2ARC - incorrect size and abnormal system load on r255173 Message-ID: <20131025061154.GA2376@hell.ukr.net> References: <1381166916.122992963.5h9ygiri@frv45.ukr.net> <1381170764.32684.31088349.343931EE@webmail.messagingengine.com> <20131007185032.GA82932@hell.ukr.net> <20131007211201.GA89306@hell.ukr.net> <20131010092223.GA28347@hell.ukr.net> <5256D9D4.2030308@allanjude.com> <20131010170334.GA71635@hell.ukr.net> <3E3F85F87B0D40C6A776A42786EEF08D@multiplay.co.uk> <20131022141055.GA34810@hell.ukr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20131022141055.GA34810@hell.ukr.net> User-Agent: Mutt/1.5.22 (2013-10-16) Cc: freebsd-current@freebsd.org, Steven Hartland , Dmitriy Makarov X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Oct 2013 06:11:58 -0000 Hello Bad news, to day two servers with applayed patches get L2ARC degraded L2 ARC Summary: (DEGRADED) Passed Headroom: 4.17m Tried Lock Failures: 635.53m IO In Progress: 41.89k Low Memory Aborts: 8 Free on Write: 1.03m Writes While Full: 12.95k R/W Clashes: 405.05k Bad Checksums: 362.19k IO Errors: 45.60k SPA Mismatch: 526.37m L2 ARC Size: (Adaptive) 391.03 GiB Header Size: 0.59% 2.30 GiB So looks like problem not disapered ^( Vitalij Satanivskij wrote: VS> VS> First of all Thank you for help. VS> VS> As for high load on system, looks like problems with l2arc have litle impact on load comparatively to another just now VS> VS> not fully classifed things. VS> VS> Looks like ower internal software and libs that it use didn't like new VMEM subsystem, at last VS> system behavior complitely diferent from 6 month older CURRENT. VS> VS> So for now none problem's with l2arc errors. VS> VS> Will try to understand reason of load and fix or at last ask for help again ^). VS> VS> VS> VS> VS> Steven Hartland wrote: VS> SH> First off I just wanted to clarify that you don't need to compression on VS> SH> dataset for L2ARC to use LZ4 compression, it does this by default as is VS> SH> not currently configurable. VS> SH> VS> SH> Next up I believe we've found the cause of this high load and I've just VS> SH> committed the fix to head: VS> SH> http://svnweb.freebsd.org/base?view=revision&sortby=file&revision=256889 VS> SH> VS> SH> Thanks to Vitalij for testing :) VS> SH> VS> SH> Dmitriy if you could test on your side too that would be appreciated. VS> SH> VS> SH> Regards VS> SH> Steve VS> SH> VS> SH> ----- Original Message ----- VS> SH> From: "Vitalij Satanivskij" VS> SH> To: "Allan Jude" VS> SH> Cc: VS> SH> Sent: Thursday, October 10, 2013 6:03 PM VS> SH> Subject: Re: ZFS L2ARC - incorrect size and abnormal system load on r255173 VS> SH> VS> SH> VS> SH> > AJ> Some background on L2ARC compression for you: VS> SH> > AJ> VS> SH> > AJ> http://wiki.illumos.org/display/illumos/L2ARC+Compression VS> SH> > VS> SH> > I'm alredy see it. VS> SH> > VS> SH> > VS> SH> > VS> SH> > AJ> http://svnweb.freebsd.org/base?view=revision&revision=251478 VS> SH> > AJ> VS> SH> > AJ> Are you sure that compression on pool/zfs is off? it would normally VS> SH> > AJ> inherit from the parent, so double check with: zfs get compression pool/zfs VS> SH> > VS> SH> > Yes, compression turned off on pool/zfs, it's was may time rechecked. VS> SH> > VS> SH> > VS> SH> > VS> SH> > AJ> Is the data on pool/zfs related to the data on the root pool? if VS> SH> > AJ> pool/zfs were a clone, and the data is actually used in both places, the VS> SH> > AJ> newer 'single copy ARC' feature may come in to play: VS> SH> > AJ> https://www.illumos.org/issues/3145 VS> SH> > VS> SH> > No, both pool and pool/zfs have diferent type of data, pool/zfs was created as new empty zfs (zfs create pool/zfs) VS> SH> > VS> SH> > and data was writed to it from another server. VS> SH> > VS> SH> > VS> SH> > Right now one machine work fine with l2arc. This machine without patch for corecting ashift on cache devices. VS> SH> > VS> SH> > At last 3 day's working with zero errors. Another servers with same config similar data, load and so on after 2 day VS> SH> > work began report abouy errors. VS> SH> > VS> SH> > VS> SH> > AJ> VS> SH> > AJ> VS> SH> > AJ> VS> SH> > AJ> -- VS> SH> > AJ> Allan Jude VS> SH> > AJ> VS> SH> > AJ> _______________________________________________ VS> SH> > AJ> freebsd-current@freebsd.org mailing list VS> SH> > AJ> http://lists.freebsd.org/mailman/listinfo/freebsd-current VS> SH> > AJ> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" VS> SH> > _______________________________________________ VS> SH> > freebsd-current@freebsd.org mailing list VS> SH> > http://lists.freebsd.org/mailman/listinfo/freebsd-current VS> SH> > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" VS> SH> > VS> SH> VS> SH> ================================================ VS> SH> This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. VS> SH> VS> SH> In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 VS> SH> or return the E.mail to postmaster@multiplay.co.uk. VS> SH> VS> SH> _______________________________________________ VS> SH> freebsd-current@freebsd.org mailing list VS> SH> http://lists.freebsd.org/mailman/listinfo/freebsd-current VS> SH> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" VS> _______________________________________________ VS> freebsd-current@freebsd.org mailing list VS> http://lists.freebsd.org/mailman/listinfo/freebsd-current VS> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"