Date: Tue, 16 Jul 2019 02:19:47 +0200 From: hw <hw@adminart.net> To: "Kevin P. Neal" <kpn@neutralgood.org> Cc: freebsd-questions@freebsd.org Subject: Re: dead slow update servers Message-ID: <87k1cij0f0.fsf@toy.adminart.net> In-Reply-To: <20190715175108.GC31450@neutralgood.org> (Kevin P. Neal's message of "Mon, 15 Jul 2019 13:51:08 -0400") References: <CAGLDxTW8zw2d%2BaBGOmBgEhipjq6ocn536fH_NScMiDD7hD=eSw@mail.gmail.com> <874l3qfvqw.fsf@toy.adminart.net> <20190714011303.GA25317@neutralgood.org> <87v9w58apd.fsf@toy.adminart.net> <f7d8acd6-6adb-2b4b-38ef-dc988d7d96a7@denninger.net> <87v9w4qjy8.fsf@toy.adminart.net> <20190715014129.GA62729@neutralgood.org> <87ftn8otem.fsf@toy.adminart.net> <20190715151621.GB31450@neutralgood.org> <87blxvjn4a.fsf@toy.adminart.net> <20190715175108.GC31450@neutralgood.org>
next in thread | previous in thread | raw e-mail | index | archive | help
"Kevin P. Neal" <kpn@neutralgood.org> writes: > On Mon, Jul 15, 2019 at 06:09:25PM +0200, hw wrote: >> "Kevin P. Neal" <kpn@neutralgood.org> writes: >> >> > On Mon, Jul 15, 2019 at 05:42:25AM +0200, hw wrote: >> >> "Kevin P. Neal" <kpn@neutralgood.org> writes: >> >> > Oh, and my Dell machines are old enough that I'm stuck with the hardware >> >> > RAID controller. I use ZFS and have raid0 arrays configured with single >> >> > drives in each. I _hate_ it. When a drive fails the machine reboots and >> >> > the controller hangs the boot until I drive out there and dump the card's >> >> > cache. It's just awful. >> >> >> >> That doesn't sound like a good setup. Usually, nothing reboots when a >> >> drive fails. >> >> >> >> Would it be a disadvantage to put all drives into a single RAID10 (or >> >> each half of them into one) and put ZFS on it (or them) if you want to >> >> keep ZFS? >> > >> > Well, it still leaves me with the overhead of dealing with creating arrays >> > in the hardware. >> >> Didn't you need to create the RAID0s having a single disk, too? > > Yes, and that wouldn't change if I took your suggestion. Which is why I > phrased it as "it still leaves" which says that the problem already exists > and would continue to exist. So it's not more work as otherwise --- maybe even less because you have only half the number of logical drives, or less. >> > And it costs me loss of the scrubbing/verification of the end-to-end >> > checksumming. So I'm less safe there with no less work. >> >> If you're worried about the controller giving results that lead to the >> correct check sums and data ending up on the disk not matching these >> check sums when the controller reads it later, what difference does it >> make which kind of RAID you use? You can always run a scrub to verify >> the check sums, and if errors are being found, you may need to replace >> the controller. > > ZFS can correct checksum errors so long as the array is still valid. Is > there a hardware RAID card that does that? I'm not sure, some controllers do what they call surface checking in the background. It's the job of the controller to make sure the data on the disk is fine, and I have no reason to assume that it doesn't do that. >> > Maybe I should just go ahead and change it. I've got a drive about to >> > fail on me. It's a three way mirror so I'm not worried about it. It would >> > be, uh, _nice_ if it didn't bring down the machine, though. >> >> If you were using two or more disks each in a RAID1 or RAID10 to create >> one disk exposed to ZFS, you wouldn't have a problem when one disk >> becomes unresponsive. If there's someone around who is used to quickly > > True, but if I run a scrub I want to verify _all_ the data on the disks, > not just the data exposed by the RAID controller card. Why? Data that is never exposed is irrelevant for this and doesn't need to be verified. Of the logical volumes you're using now, the controller card exposes no more than it does, so what difference does it make of how many disks the volume is made of? You don't get any more or less data exposed as the controller does its thing regardless. Besides, how do you suggest to get all data exposed? Do you have special firmware on your disks that exposes all data, and for everything else that is involved? > That means ZFS needs to see _all_ the disks. Otherwise I would be > vulnerable to loss of data due to checksum errors that may only be > seen after a drive dies. At that point it's too late to correct. I don't understand how it would make a difference whether ZFS can see all physical disks or not. One way or another, there is merely a storage device ZFS is using and can see, and if the device is errorneous, it shouldn't matter what the device is physically made of. When a disk fails and is replaced, the hardware RAID is being rebuilt, and when ZFS detects checksum errors on that storage device, why shouldn't it be able to correct the errors like with any other storage device? It's even not the storage device that failed but only a disk. What if you had hybrid disks consisting of some flash memory and magnetic disks? You would have to conclude that they can never be used with ZFS because they could introduce hidden checksum errors. What about the cache of your disks, do you turn it off because hidden checksum errors could be introduced when there is a mismatch between the data on the disk and what is being exposed by the cache? IIRC, one of the advantages of ZFS is that you don't need to disable the cache. RAID controllers and such aren't designed to create hidden checksum errors ZFS is unable to correct. ZFS was designed to detect errors and to correct them, using check sums. There's no need to be afraid of RAID controllers and such. >> Hardware RAID does have advantages, so why not use them when you're >> stuck with it anyway? > > Because the downsides outweigh the upside. Like how? The whole setup seems very questionable, and the machine goes out of service right away when a disk becomes unresponsive. One of the purposes of RAID is to prevent just that.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?87k1cij0f0.fsf>