Date: Tue, 9 Apr 2019 17:27:29 -0400 From: Zaphod Beeblebrox <zbeeble@gmail.com> To: Karl Denninger <karl@denninger.net> Cc: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) Message-ID: <CACpH0MdLNQ_dqH%2Bto=amJbUuWprx3LYrOLO0rQi7eKw-ZcqWJw@mail.gmail.com> In-Reply-To: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> References: <f87f32f2-b8c5-75d3-4105-856d9f4752ef@denninger.net> <c96e31ad-6731-332e-5d2d-7be4889716e1@FreeBSD.org> <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
I have a "Ghetto" home RAID array. It's built on compromises and makes use of RAID-Z2 to survive. It consists of two plexes of 8x 4T units of "spinning rust". It's been upgraded and upgraded. It started as 8x 2T, then 8x 2T + 8x 4T then the current 16x 4T. The first 8 disks are connected to motherboard SATA. IIRC, there are 10. Two ports are used for a mirror that it boots from. There's also an SSD in there somhow, so it might be 12 ports on the motherboard. The other 8 disks started life in eSATA port multiplier boxes. That was doubleplusungood, so I got a RAID card based on LSI pulled from a fujitsu server in Japan. That's been upgraded a couple of times... not always a good experience. One problem is that cheap or refurbished drives don't always "like" SAS controllers and FreeBSD. YMMV. Anyways, this is all to introduce the fact that I've seen this behaviour multiple times. You have a drive that leaves the array for some amount of time, and after resilvering, a scrub will find a small amount of bad data. 32 k or 40k or somesuch. In my cranial schema of things, I've chalked it up to out-of-order writing of the drives ... or other such behavior s.t. ZFS doesn't know exactly what has been written. I've often wondered if the fix would be to add an amount of fuzz to the transaction range that is resilvered. On Tue, Apr 9, 2019 at 4:32 PM Karl Denninger <karl@denninger.net> wrote: > On 4/9/2019 15:04, Andriy Gapon wrote: > > On 09/04/2019 22:01, Karl Denninger wrote: > >> the resilver JUST COMPLETED with no errors which means the ENTIRE DISK'S > >> IN USE AREA was examined, compared, and blocks not on the "new member" > >> or changed copied over. > > I think that that's not entirely correct. > > ZFS maintains something called DTL, a dirty-time log, for a missing / > offlined / > > removed device. When the device re-appears and gets resilvered, ZFS > walks only > > those blocks that were born within the TXG range(s) when the device was > missing. > > > > In any case, I do not have an explanation for what you are seeing. > > That implies something much more-serious could be wrong such as given > enough time -- a week, say -- that the DTL marker is incorrect and some > TXGs that were in fact changed since the OFFLINE are not walked through > and synchronized. That would explain why it gets caught by a scrub -- > the resilver is in fact not actually copying all the blocks that got > changed and so when you scrub the blocks are not identical. Assuming > the detached disk is consistent that's not catastrophically bad IF > CAUGHT; where you'd get screwed HARD is in the situation where (for > example) you had a 2-unit mirror, detached one, re-attached it, resilver > says all is well, there is no scrub performed and then the > *non-detached* disk fails before there is a scrub. In that case you > will have permanently destroyed or corrupted data since the other disk > is allegedly consistent but there are blocks *missing* that were never > copied over. > > Again this just showed up on 12.x; it definitely was *not* at issue in > 11.1 at all. I never ran 11.2 in production for a material amount of > time (I went from 11.1 to 12.0 STABLE after the IPv6 fixes were posted > to 12.x) so I don't know if it is in play on 11.2 or not. > > I'll see if it shows up again with 20.00.07.00 card firmware. > > Of note I cannot reproduce this on my test box with EITHER 19.00.00.00 > or 20.00.07.00 firmware when I set up a 3-unit mirror, offline one, make > a crap-ton of changes, offline the second and reattach the third (in > effect mirroring the "take one to the vault" thing) with a couple of > hours elapsed time and a synthetic (e.g. "dd if=/dev/random of=outfile > bs=1m" sort of thing) "make me some new data that has to be resilvered" > workload. I don't know if that's because I need more entropy in the > filesystem than I can reasonably generate this way (e.g. more > fragmentation of files, etc) or whether it's a time-based issue (e.g. > something's wrong with the DTL/TXG thing as you note above in terms of > how it functions and it only happens if the time elapsed causes > something to be subject to a rollover or similar problem.) > > I spent quite a lot of time trying to make reproduce the issue on my > "sandbox" machine and was unable -- and of note it is never a large > quantity of data that is impacted, it's usually only a couple of dozen > checksums that show as bad and fixed. Of note it's also never just one; > if there was a single random hit on a data block due to ordinary bitrot > sort of issues I'd expect only one checksum to be bad. But generating a > realistic synthetic workload over the amount of time involved on a > sandbox is not trivial at all; the system on which this is now happening > handles a lot of email and routine processing of various sorts including > a fair bit of database activity associated with network monitoring and > statistical analysis. > > I'm assuming that using "offline" as a means to do this hasn't become > "invalid" as something that's considered "ok" as a means of doing this > sort of thing.... it certainly has worked perfectly well for a very long > time! > > -- > Karl Denninger > karl@denninger.net <mailto:karl@denninger.net> > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACpH0MdLNQ_dqH%2Bto=amJbUuWprx3LYrOLO0rQi7eKw-ZcqWJw>