Date: Wed, 16 Oct 2013 18:57:16 +0300 From: Vitalij Satanivskij <satan@ukr.net> To: Steven Hartland <killing@multiplay.co.uk> Cc: Vitalij Satanivskij <satan@ukr.net>, Dmitriy Makarov <supportme@ukr.net>, "Justin T. Gibbs" <gibbs@freebsd.org>, Borja Marcos <borjam@sarenet.es>, freebsd-current@freebsd.org Subject: Re: ZFS secondarycache on SSD problem on r255173 Message-ID: <20131016155716.GA90462@hell.ukr.net> In-Reply-To: <6DE320B20F7844B9ADC34A214AED8055@multiplay.co.uk> References: <1379333192.127359970.ma5jnbc5@fmst-6.ukr.net> <1379334340.567465877.0b1lli6r@fmst-6.ukr.net> <8365CE736DC749DF95D0030A725211F6@multiplay.co.uk> <02549AD9-C456-4E17-927C-B4BCC97F8CC8@freebsd.org> <4AA28730F331444AB13108ABF0CD68B7@multiplay.co.uk> <1379496242.750778745.m0ksff1m@fmst-6.ukr.net> <20131016080100.GA27758@hell.ukr.net> <0023A5B16E614B67A2D8A4A63641D1D8@multiplay.co.uk> <20131016141053.GA43384@hell.ukr.net> <6DE320B20F7844B9ADC34A214AED8055@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Steven Hartland wrote:
SH> I'm not clear what you rolled back there as r255173 has ntothing to do
SH> with this. Could you clarify
r255173 with you patch from email dated Tue, 17 Sep 2013 23:53:12 +0100 with subject Re: ZFS secondarycache on SSD problem on r255173
Errors wich we gets is in arcstats count not in messages, and was desribed some time ago in mails
be me and Dmitriy Makarov with subject's ZFS L2ARC - incorrect size and abnormal system load on r255173
On r255173 without patch and with vfs.zfs.max_auto_ashift=9 when added to pool 2 ssd as caches get
cache
gpt/cache1 ONLINE 0 0 0 block size: 512B configured, 4096B native
gpt/cache2 ONLINE 0 0 0 block size: 512B configured, 4096B native
Same message we seen with default vfs.zfs.max_auto_ashift
Will wait some time to see how it works.
SH> Any errors recorded in /var/log/messages?
SH>
SH> Could you add code to record the none zero value of zio->io_error in
SH> l2arc_read_done as this may give some indication of the underlying
SH> issue.
SH>
SH> Additionally could always put a panic in that code path too and then
SH> create a dump so the details can be fully exhamined.
SH>
SH> In terms of the slowness thats going to be a side effect of the cache
SH> failures.
SH>
SH> Oh could you also confirm that the issue doesn't exist if you
SH> 1. Exclude r255753
SH> 2. Set vfs.zfs.max_auto_ashift=9
SH>
SH> Regards
SH> Steve
SH> ----- Original Message -----
SH> From: "Vitalij Satanivskij" <satan@ukr.net>
SH> To: "Steven Hartland" <killing@multiplay.co.uk>
SH> Cc: "Vitalij Satanivskij" <satan@ukr.net>; "Dmitriy Makarov" <supportme@ukr.net>; "Justin T. Gibbs" <gibbs@freebsd.org>; "Borja
SH> Marcos" <borjam@sarenet.es>; <freebsd-current@freebsd.org>
SH> Sent: Wednesday, October 16, 2013 3:10 PM
SH> Subject: Re: ZFS secondarycache on SSD problem on r255173
SH>
SH>
SH> > Yes
SH> >
SH> > We have 15 servers, all of them have problem while using with patch fo ashift, sh we rollback path (for r255173)
SH> > and all of them works for a week without that's problem's. Yesterday one of of servers was updated to stable/10 (beta1)
SH> >
SH> > wich include patch and after around 12 hours of works l2arc begin et errors like that
SH> >
SH> > kstat.zfs.misc.arcstats.l2_cksum_bad
SH> > kstat.zfs.misc.arcstats.l2_io_error
SH> >
SH> >
SH> > For now patch disabled in ower production.
SH> >
SH> >
SH> > Please note we have very heavy load on zfs pool so 90GB arc and 3x180Gb L2arc have very big hit's on it on it.
SH> >
SH> >
SH> > SSD used for cache's is intel ssd 530 series smart for all devices in in normal states's
SH> > no bad values on it.
SH> >
SH> > Steven Hartland wrote:
SH> > SH> Have you confirmed the ashift changes are the actual cause of this
SH> > SH> by backing out just those changes and retesting on the same hardware.
SH> > SH>
SH> > SH> Also worth checking your disks smart values to confirm there are no
SH> > SH> visible signs of HW errors.
SH> > SH>
SH> > SH> Regards
SH> > SH> Steve
SH> > SH>
SH> > SH> ----- Original Message -----
SH> > SH> From: "Vitalij Satanivskij" <satan@ukr.net>
SH> > SH> To: "Dmitriy Makarov" <supportme@ukr.net>
SH> > SH> Cc: "Steven Hartland" <killing@multiplay.co.uk>; "Justin T. Gibbs" <gibbs@freebsd.org>; "Borja Marcos" <borjam@sarenet.es>;
SH> > SH> <freebsd-current@freebsd.org>
SH> > SH> Sent: Wednesday, October 16, 2013 9:01 AM
SH> > SH> Subject: Re: ZFS secondarycache on SSD problem on r255173
SH> > SH>
SH> > SH>
SH> > SH> > Hello.
SH> > SH> >
SH> > SH> > Patch brocke cache functionality.
SH> > SH> >
SH> > SH> > Look at's Dmitriy's mail from Mon, 07 Oct 2013 21:09:06 +0300
SH> > SH> >
SH> > SH> > With subject ZFS L2ARC - incorrect size and abnormal system load on r255173
SH> > SH> >
SH> > SH> > As patch alredy in head and BETA it's not good.
SH> > SH> >
SH> > SH> > Yesterday we update one machine up to beta1 and forgot about patch. So 12 Hours and cache broken... :((
SH> > SH> >
SH> > SH> >
SH> > SH> >
SH> > SH> > Dmitriy Makarov wrote:
SH> > SH> > DM> The attached patch by Steven Hartland fixes issue for me too. Thank you!
SH> > SH> > DM>
SH> > SH> > DM>
SH> > SH> > DM> --- Исходное сообщение ---
SH> > SH> > DM> От кого: "Steven Hartland" < killing@multiplay.co.uk >
SH> > SH> > DM> Дата: 18 сентября 2013, 01:53:10
SH> > SH> > DM>
SH> > SH> > DM> ----- Original Message -----
SH> > SH> > DM> From: "Justin T. Gibbs" <
SH> > SH> > DM>
SH> > SH> > DM> ---
SH> > SH> > DM> Дмитрий Макаров
SH> > SH> > DM> _______________________________________________
SH> > SH> > DM> freebsd-current@freebsd.org mailing list
SH> > SH> > DM> http://lists.freebsd.org/mailman/listinfo/freebsd-current
SH> > SH> > DM> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
SH> > SH> >
SH> > SH>
SH> > SH>
SH> > SH> ================================================
SH> > SH> This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the
SH> > event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any
SH> > information contained in it.
SH> > SH>
SH> > SH> In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
SH> > SH> or return the E.mail to postmaster@multiplay.co.uk.
SH> > SH>
SH> > SH> _______________________________________________
SH> > SH> freebsd-current@freebsd.org mailing list
SH> > SH> http://lists.freebsd.org/mailman/listinfo/freebsd-current
SH> > SH> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
SH> >
SH>
SH>
SH> ================================================
SH> This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
SH>
SH> In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
SH> or return the E.mail to postmaster@multiplay.co.uk.
SH>
SH> _______________________________________________
SH> freebsd-current@freebsd.org mailing list
SH> http://lists.freebsd.org/mailman/listinfo/freebsd-current
SH> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131016155716.GA90462>