From owner-freebsd-stable@freebsd.org Fri Jun 7 00:50:01 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F78915C456A for ; Fri, 7 Jun 2019 00:50:01 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (hades.sorbs.net [72.12.213.40]) by mx1.freebsd.org (Postfix) with ESMTP id D64E88E361 for ; Fri, 7 Jun 2019 00:49:58 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (gate.mhix.org [203.206.128.220]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0PSP00BY4C6UO820@hades.sorbs.net> for freebsd-stable@freebsd.org; Thu, 06 Jun 2019 17:04:08 -0700 (PDT) Subject: Re: ZFS... From: Michelle Sullivan To: Steven Hartland Cc: freebsd-stable References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <70fac2fe3f23f85dd442d93ffea368e1@ultra-secure.de> <70C87D93-D1F9-458E-9723-19F9777E6F12@sorbs.net> <5ED8BADE-7B2C-4B73-93BC-70739911C5E3@sorbs.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> <34539589-162B-4891-A68F-88F879B59650@sorbs.net> <576857a5-a5ab-eeb8-2391-992159d9c4f2@denninger.net> <7DBA7907-BE8F-4944-9A71-86E5AC1B85CA@gromit.dlib.vt.edu> <5c458075-351f-6eb6-44aa-1bd268398343@sorbs.net> <289FE04E-1692-4763-96B3-91E8C1BBBBD6@sorbs.net> Message-id: <70d5e0d3-c188-7f06-cf53-702d565fd481@sorbs.net> Date: Fri, 07 Jun 2019 09:49:48 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:51.0) Gecko/20100101 Firefox/51.0 SeaMonkey/2.48 In-reply-to: <289FE04E-1692-4763-96B3-91E8C1BBBBD6@sorbs.net> X-Rspamd-Queue-Id: D64E88E361 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of michelle@sorbs.net designates 72.12.213.40 as permitted sender) smtp.mailfrom=michelle@sorbs.net X-Spamd-Result: default: False [-1.74 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.953,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:hades.sorbs.net]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sorbs.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[battlestar.sorbs.net,anaconda.sorbs.net,ninja.sorbs.net,catapilla.sorbs.net,scorpion.sorbs.net,desperado.sorbs.net]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.213.12.72.list.dnswl.org : 127.0.10.0]; SUBJ_ALL_CAPS(0.45)[6]; IP_SCORE(-0.29)[ip: (-0.71), ipnet: 72.12.192.0/19(-0.37), asn: 11114(-0.29), country: US(-0.06)]; NEURAL_HAM_SHORT(-0.25)[-0.245,0]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11114, ipnet:72.12.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; CTE_CASE(0.50)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jun 2019 00:50:01 -0000 Michelle Sullivan wrote: >> On 02 May 2019, at 03:39, Steven Hartland wrote: >> >> >> >>> On 01/05/2019 15:53, Michelle Sullivan wrote: >>> Paul Mather wrote: >>>>> On Apr 30, 2019, at 11:17 PM, Michelle Sullivan wrote: >>>>> >>>>> Been there done that though with ext2 rather than UFS.. still got all my data back... even though it was a nightmare.. >>>> >>>> Is that an implication that had all your data been on UFS (or ext2:) this time around you would have got it all back? (I've got that impression through this thread from things you've written.) That sort of makes it sound like UFS is bulletproof to me. >>> Its definitely not (and far from it) bullet proof - however when the data on disk is not corrupt I have managed to recover it - even if it has been a nightmare - no structure - all files in lost+found etc... or even resorting to r-studio in the even of lost raid information etc.. >> Yes but you seem to have done this with ZFS too, just not in this particularly bad case. >> > There is no r-studio for zfs or I would have turned to it as soon as this issue hit. > > > So as an update, this Company: http://www.klennet.com/ produce a ZFS recovery tool: https://www.klennet.com/zfs-recovery/default.aspx and following several code changes due to my case being an 'edge case' the entire volume (including the zvol - which I previously recovered as it wasn't suffering from the metadata corruption) and all 34 million files is being recovered intact with the entire directory structure. Its only drawback is it's a windows only tool, so I built 'windows on a stick' and it's running from that. The only thing I had to do was physically pull the 'spare' out as the spare already had data on it from being previously swapped in and it confused the hell out of the algorithm that detects the drive order. Regards, Michelle -- Michelle Sullivan http://www.mhix.org/