From owner-freebsd-stable@freebsd.org Wed May 8 22:55:33 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15A301595ADE for ; Wed, 8 May 2019 22:55:33 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (hades.sorbs.net [72.12.213.40]) by mx1.freebsd.org (Postfix) with ESMTP id 4DCBD6B44B for ; Wed, 8 May 2019 22:55:32 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Received: from [10.10.0.230] (gate.mhix.org [203.206.128.220]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0PR70018VKBSZI70@hades.sorbs.net> for freebsd-stable@freebsd.org; Wed, 08 May 2019 16:09:31 -0700 (PDT) Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and truncated. Subject: Re: ZFS... From: Michelle Sullivan X-Mailer: iPad Mail (16A404) In-reply-to: Date: Thu, 09 May 2019 08:55:28 +1000 Cc: freebsd-stable@freebsd.org Content-transfer-encoding: quoted-printable Message-id: <7D18A234-E7BF-4855-BD51-4AE2253DB1E4@sorbs.net> References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <56833732-2945-4BD3-95A6-7AF55AB87674@sorbs.net> <3d0f6436-f3d7-6fee-ed81-a24d44223f2f@netfence.it> <17B373DA-4AFC-4D25-B776-0D0DED98B320@sorbs.net> <70fac2fe3f23f85dd442d93ffea368e1@ultra-secure.de> <70C87D93-D1F9-458E-9723-19F9777E6F12@sorbs.net> <5ED8BADE-7B2C-4B73-93BC-70739911C5E3@sorbs.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <20190430102024.E84286@mulder.mintsol.com> <41FA461B-40AE-4D34-B280-214B5C5868B5@punkt.de> <20190506080804.Y87441@mulder.mintsol.com> <08E46EBF-154F-4670-B411-482DCE6F395D@sorbs.net> <33D7EFC4-5C15-4FE0-970B-E6034EF80BEF@gromit.dlib.vt.edu> To: Walter Parker X-Rspamd-Queue-Id: 4DCBD6B44B X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of michelle@sorbs.net designates 72.12.213.40 as permitted sender) smtp.mailfrom=michelle@sorbs.net X-Spamd-Result: default: False [-2.09 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.968,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:hades.sorbs.net]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sorbs.net]; NEURAL_HAM_SHORT(-0.03)[-0.030,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: battlestar.sorbs.net]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.213.12.72.list.dnswl.org : 127.0.10.0]; SUBJ_ALL_CAPS(0.45)[6]; IP_SCORE(-0.33)[ip: (-0.82), ipnet: 72.12.192.0/19(-0.43), asn: 11114(-0.34), country: US(-0.06)]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11114, ipnet:72.12.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 May 2019 22:55:33 -0000 Michelle Sullivan http://www.mhix.org/ Sent from my iPad On 09 May 2019, at 01:55, Walter Parker wrote: >>=20 >>=20 >> ZDB (unless I'm misreading it) is able to find all 34m+ files and >> verifies the checksums. The problem is in the zfs data structures (one >> definitely, two maybe, metaslabs fail checksums preventing the mounting >> (even read-only) of the volumes.) >>=20 >>> Especially, how to you know >>> before you recovered the data from the drive. >> See above. >>=20 >>> As ZFS meta data is stored >>> redundantly on the drive and never in an inconsistent form (that is what= >>> fsck does, it fixes the inconsistent data that most other filesystems >> store >>> when they crash/have disk issues). >> The problem - unless I'm reading zdb incorrectly - is limited to the >> structure rather than the data. This fits with the fact the drive was >> isolated from user changes when the drive was being resilvered so the >> data itself was not being altered .. that said, I am no expert so I >> could easily be completely wrong. >>=20 >> What it sounds like you need is a meta data fixer, not a file recovery > tool. This is true, but I am of the thought in alignment with the zFs devs this mi= ght not be a good idea... if zfs can=E2=80=99t work it out already, the best= thing to do will probably be get everything off it and reformat... =20 > Assuming the meta data can be fixed that would be the easy route. That=E2=80=99s the thing... I don=E2=80=99t know if it can be easily fixed..= . more I think the meta data can probably be easily fixed, but I suspect the= spacemap can=E2=80=99t and as such if it can=E2=80=99t there is going to be= one of two things... either a big hole (or multiple little ones) or the li= kelihood of new data overwriting partially or in full, old data and this wou= ld not be good.. > That sound not be hard to write if everything else on the disk has no > issues. Don't you say in another message that the system is now returning > 100's of drive errors. No, one disk in the 16 disk zRAID2 ... previously unseen but it could be th= e errors have occurred in the last 6 weeks... everytime I reboot it started r= esilvering, gets to 761M resilvered and then stops. > How does that relate the statement =3D>Everything on > the disk is fine except for a little bit of corruption in the freespace ma= p? Well I think it goes through until it hits that little bit of corruption at s= tops it mounting... then stops again.. I=E2=80=99m seeing 100s of hard errors at the beginning of one of the drives= .. they were reported in syslog but only just so could be a new thing. Coul= d be previously undetected.. no way to know. >=20 >=20 >>=20 >>>=20 >>> I have a friend/business partner that doesn't want to move to ZFS becaus= e >>> his recovery method is wait for a single drive (no-redundancy, sometimes= >> no >>> backup) to fail and then use ddrescue to image the broken drive to a new= >>> drive (ignoring any file corruption because you can't really tell withou= t >>> ZFS). He's been using disk rescue programs for so long that he will not >>> move to ZFS, because it doesn't have a disk rescue program. >>=20 >> The first part is rather cavilier .. the second part I kinda >> understand... its why I'm now looking at alternatives ... particularly >> being bitten as badly as I have with an unmountable volume. >>=20 >> On the system I managed for him, we had a system with ZFS crap out. I > restored it from a backup. I continue to believe that people running > systems without backups are living on borrowed time. The idea of relying o= n > a disk recovery tool is too risky for my taste. >=20 >=20 >>> He has systems >>> on Linux with ext3 and no mirroring or backups. I've asked about moving >>> them to a mirrored ZFS system and he has told me that the customer >> doesn't >>> want to pay for a second drive (but will pay for hours of his time to fi= x >>> the problem when it happens). You kind of sound like him. >> Yeah..no! I'd be having that on a second (mirrored) drive... like most >> of my production servers. >>=20 >>> ZFS is risky >>> because there isn't a good drive rescue program. >> ZFS is good for some applications. ZFS is good to prevent cosmic ray >> issues. ZFS is not good when things go wrong. ZFS doesn't usually go >> wrong. Think that about sums it up. >>=20 >> When it does go wrong I restore from backups. Therefore my systems don't > have problems. I sorry you had the perfect trifecta that caused you to los= e > multiple drives and all your backups at the same time. >=20 >=20 >>> Sun's design was that the >>> system should be redundant by default and checksum everything. If the >>> drives fail, replace them. If they fail too much or too fast, restore >> from >>> backup. Once the system had too much corruption, you can't recover/check= >>> for all the damage without a second off disk copy. If you have that off >>> disk, then you have backup. They didn't build for the standard use case >> as >>> found in PCs because the disk recover programs rarely get everything >> back, >>> therefore they can't be relied on to get you data back when your data is= >>> important. Many PC owners have brought PC mindset ideas to the "UNIX" >>> world. Sun's history predates Windows and Mac and comes from a >>> Mini/Mainframe mindset (were people tried not to guess about data >>> integrity). >> I came from the days of Sun. >>=20 >> Good then you should understand Sun's point of view. >=20 >=20 >>>=20 >>> Would a disk rescue program for ZFS be a good idea? Sure. Should the lac= k >>> of a disk recovery program stop you from using ZFS? No. If you think so,= >> I >>> suggest that you have your data integrity priorities in the wrong order >>> (focusing on small, rare events rather than the common base case). >> Common case in your assessment in the email would suggest backups are >> not needed unless you have a rare event of a multi-drive failure. Which >> I know you're not advocating, but it is this same circular argument... >> ZFS is so good it's never wrong we don't need no stinking recovery >> tools, oh but take backups if it does fail, but it won't because it's so >> good and you have to be running consumer hardware or doing something >> wrong or be very unlucky with failures... etc.. round and round we go, >> where ever she'll stop no-one knows. >>=20 >> I advocate 2-3 backups of any important system (at least one different > that the other, offsite if one can afford it). > I never said ZFS is so good we don't need backups (that would be a stupid > comment). As far as a recovery tool, those sound risky. I'd prefer > something without so much risk. >=20 > Make your own judgement, it is your time and data. I think ZFS is a great > filesystem that anyone using FreeBSD or Illumios should be using. >=20 >=20 > --=20 > The greatest dangers to liberty lurk in insidious encroachment by men of > zeal, well-meaning but without understanding. -- Justice Louis D. Brande= is > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"