Date: Mon, 22 Jun 2015 01:23:30 +0200 From: Willem Jan Withagen <wjw@digiware.nl> To: Quartz <quartz@sneakertech.com> Cc: freebsd-fs@freebsd.org Subject: Re: This diskfailure should not panic a system, but just disconnect disk from ZFS Message-ID: <55874772.4090607@digiware.nl> In-Reply-To: <55871F4C.5010103@sneakertech.com> References: <5585767B.4000206@digiware.nl> <558590BD.40603@isletech.net> <5586C396.9010100@digiware.nl> <55871F4C.5010103@sneakertech.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 21/06/2015 22:32, Quartz wrote: >> Or do I have to high hopes of ZFS? >> And is a hung disk a 'catastrophic pool failure'? > > Yes to both. > > I encountered this exact same issue a couple years ago (and complained > about it to this list as well, although I didn't get a complete answer > at the time. I can provide links to the conversation if interested). > > Basically, the heart of the issue is the way the kernel/drivers/ZFS > deals with IO and DMA. There's currently no way to tell what's going on > with the disks and what outstanding IO to the pool can be dropped or > ignored. As-currently-designed there's no safe way to just kick out the > pool and keep going, so the only options are to wait, panic, or wait and > then panic. Fixing this would require a major rewrite of a lot of code, > which isn't going to happen any time soon. The failmode setting and > deadman timer were implemented as a bandage to prevent the system from > hanging forever. > > See this page for more info: > http://comments.gmane.org/gmane.os.illumos.zfs/61 > Yes, I know of this discussion already as long as I'm working with ZFS. But reading it like this does make it much more sense. Perhaps the text should suggest some thing more painfull :), since now it suggest that chopping the disk would help... (Hence my reaction) But especially the hung disk during reading is sort of a ticking timebomb... On the other hand, if it is already outstanding for 100's of seconds, then it is never going to arrive. So the chance of running into corrupted memory is going to be 0.00000...... But I do agree that in this case a panic might be next best solution. >From another response I conclude that htere could be something in the driver/hardware combo that could run me into trouble... >> All failmode settings result in a seriously handicapped system... > > Yes. Again, this is a design issue/flaw with how DMA works. There's no > real way to continue on gracefully when a pool completely dies due to > hung IO. > > We're all pretty much stuck with this problem, at least for quite a while. We'll the pool did not die, (at least not IMHO) just one disk stopt working.... The pool is still resilient, so it could continue alert the operator have the operator fix/reboot/..... But it the fact that a stalled DMA action could "corrupt" memory after the fact. Just because a command outstanding for way too long, does all of a sudden completes. >> Is waiting only meant to wait a limited time? And then panic anyways? > > By default yes. However, if you know that on your system the issue will > eventually resolve itself given several hours (and you want to wait that > long) you can change the deadman timeout or disable it completely. Look > at "vfs.zfs.deadman_enabled" and "vfs.zfs.deadman_synctime". I see: vfs.zfs.deadman_enabled: 1 vfs.zfs.deadman_checktime_ms: 5000 vfs.zfs.deadman_synctime_ms: 1000000 So the "hung" on this I/O action has taken 1000 secs, and did not complete... I guess that if I like to live dangerously, I could set enabled to 0, and run the risk... ?? Probably the better solution is to see if it occurs more often, and in that case upgrade to modern hardware with newer HD controllers. Current config has worked for me for already quite some time. For the time being I'll offload a few disks to a mvs controller that is the only PCIe slot in the MB. Thanx for all the info, --WjW
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55874772.4090607>