From owner-freebsd-fs@freebsd.org Tue Mar 19 13:14:46 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE2191550B6C for ; Tue, 19 Mar 2019 13:14:46 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (hades.sorbs.net [72.12.213.40]) by mx1.freebsd.org (Postfix) with ESMTP id 437DD84E03 for ; Tue, 19 Mar 2019 13:14:45 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (gate.mhix.org [203.206.128.220]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0POM0018O82WUY00@hades.sorbs.net> for freebsd-fs@freebsd.org; Tue, 19 Mar 2019 06:28:10 -0700 (PDT) To: "freebsd-fs@freebsd.org" From: Michelle Sullivan Subject: zfs corruption (again) due to interupted resilver and power faults. Message-id: Date: Wed, 20 Mar 2019 00:14:39 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:51.0) Gecko/20100101 Firefox/51.0 SeaMonkey/2.48 X-Rspamd-Queue-Id: 437DD84E03 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of michelle@sorbs.net designates 72.12.213.40 as permitted sender) smtp.mailfrom=michelle@sorbs.net X-Spamd-Result: default: False [-1.16 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.91)[-0.909,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:hades.sorbs.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[sorbs.net]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.99)[-0.992,0]; IP_SCORE(-0.01)[country: US(-0.07)]; NEURAL_SPAM_SHORT(0.47)[0.467,0]; MX_GOOD(-0.01)[cached: battlestar.sorbs.net]; RCVD_IN_DNSWL_NONE(0.00)[40.213.12.72.list.dnswl.org : 127.0.10.0]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; CTE_CASE(0.50)[]; ASN(0.00)[asn:11114, ipnet:72.12.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 13:14:46 -0000 So lost my pool again... this time looks more hopeless, but for simpler reasons. Basically a significant power issue occurred whilst in a resilver process which I am lucky no-one was killed - though if I find the idiot there is still time... anyhoo, to the issue at hand... FreeBSD 12 (memstick boot) Can't import the pool 'storage' All devices are now online (one wasn't but that issue has been resolved by byte copying the drive to a new drive and inserting) Its a zraid2 set... only the one drive had failed and was in the middle of the resilver when the power issue occurred. Pool refuses to import (even with -FfX) zdb reports 4 metdata errors and no (zero) data errors... see: http://flashback.sorbs.net/packages/zfs/image4.jpeg 'zdb -bcd' asserts out in space_map_load (123 of 163) see: http://flashback.sorbs.net/packages/zfs/image5.jpeg Thoughts on how I might recover this? (Really need to as was in the middle of blowing away the original (backup) server for a rebuild and new backup.. so I have quite literally no backups, and every photo I have taken are in that zvol on the pool - there should be no reason why it's corrupt as with the exception of the resilver - there was no active writing to the pool.) Yours hopefully, Michelle -- Michelle Sullivan http://www.mhix.org/