From owner-freebsd-questions@freebsd.org Mon Feb 1 23:42:13 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CE2694EB6F8 for ; Mon, 1 Feb 2021 23:42:13 +0000 (UTC) (envelope-from matt@gsicomp.on.ca) Received: from outbound1a.ore.mailhop.org (outbound1a.ore.mailhop.org [54.213.22.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DV4H86MQvz3PkB for ; Mon, 1 Feb 2021 23:42:12 +0000 (UTC) (envelope-from matt@gsicomp.on.ca) ARC-Seal: i=1; a=rsa-sha256; t=1612222931; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=fcS8+3VBTr99dqAKgqyLRTs+65LUoG3FeL7pEWqmay23Cuu8CUC6LhLHwAD8FruBuw1rbNasZ2oLc X8lFA/WslcQBmdJsySg+230LSzaNA1r+7a4KFI+tShaMUVXv/gddkgh9NO5Gz2qBLtORdH6IAdO0SJ AIADpcfTXwkh1LqbVMStoLrE76NkskBAnQKW6q4NvbvYTzYMAoUCxKJePSbuXgSBLzPKgUz2TJvGiG /rUCvJXalDR/7UanTmEdLd/DS26dl6iV6h60eMxLVNIEj4wHvQXjP/LyXswk8aeCp7sq33l0YuAoYi gHFgErNzBwP3RuewTsg2xRnOg35A3mQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:content-type:mime-version:message-id:date:subject: in-reply-to:references:cc:to:from:dkim-signature:dkim-signature:from; bh=L6eE6ZOqlbaK6WA+lqN5pc0YR6Aq6qsJfdLFq5QShNw=; b=fkiRByPmu+tt+x036/VJNJ4uDuwmY8xWPd7HkpMLAPBzpi4g5KAzA3XmcTZLExEc9Egz4CXrEPgW7 twBJS9HctuwXopXAOuYd/c70DC38+8kQYps16AjASnU2velk31ALJceKLjABzAsBKPuhToyHSRezWM z2l4rn11yRhrMp8rpzRqukWTiJl5EN4pdgadcNO7f7ztYQrcjatGFqO7pxDCh3guA0nO3TMXHv9eSs BKSHZLzzOqw1USWocg00rT8HmpkqIAA63IK6oc8ig9bkmh9qBZ4p5vcJusk+zPqnb+sXNy4k5Lxth4 bSNfBCJFobonSPL7IRHnBCA3wSRqKFA== ARC-Authentication-Results: i=1; outbound3.ore.mailhop.org; spf=pass smtp.mailfrom=gsicomp.on.ca smtp.remote-ip=162.243.98.91; dmarc=none header.from=gsicomp.on.ca; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gsicomp.on.ca; s=duo-1597798936401-95ae6877; h=content-transfer-encoding:content-type:mime-version:message-id:date:subject: in-reply-to:references:cc:to:from:from; bh=L6eE6ZOqlbaK6WA+lqN5pc0YR6Aq6qsJfdLFq5QShNw=; b=yrnTo/RP8a/g22FSHrAxiFObSZJuecd0IEnTbUWSjxvqmh9brFEt/3zk1JDFvH8dWdPEP18mXd6FB XeN9WszcXuDTHfdYMT8dZBlodMEniSV7k9cxnHdvvvx2AtrqyhO+yh2Sx4Gd42sHG5uGMlx6OVsqg0 vYQrQleixtshlvc8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:content-type:mime-version:message-id:date:subject: in-reply-to:references:cc:to:from:from; bh=L6eE6ZOqlbaK6WA+lqN5pc0YR6Aq6qsJfdLFq5QShNw=; b=u/ZhR3x+ZrbVvNY5e5Qxp2tWGJjgUri5sjfRJ8du5Xq6hvM1m/VQDhELgAZooguCFgJuFWsF6BGxA eR3hcLgp392puokIxUTGEBDK1yeRvCCuUeo2tWACdLZ5T5316QgAEWHZN6kWxrd9DZdsYd5WovU1VV 1Nn/x5MtDL9X3Ch5tO0LIcjIBwo4aBAewnTWTZfDm9Q81CUOl5exrQ6STU2J72X2ML+fQWisTGetpT WwNVzg8C3Gt3U8OPLN3zV68rNDyCBv9OOTVVb3gKd4xPinTsaZ6RfRZKuW0h2URxuf/IEDEYQ6ad9v GOUvC0vD3fwyctHhdSoi3H1XtISCZjw== X-Originating-IP: 162.243.98.91 X-MHO-RoutePath: bWVtbWVydG8= X-MHO-User: 194f5f67-64e7-11eb-9805-2759e409b1e0 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Mail-Handler: DuoCircle Outbound SMTP Received: from mail.salamander.gsicomp.on.ca (salamander.gsicomp.on.ca [162.243.98.91]) by outbound3.ore.mailhop.org (Halon) with ESMTPSA id 194f5f67-64e7-11eb-9805-2759e409b1e0; Mon, 01 Feb 2021 23:42:10 +0000 (UTC) Received: by mail.salamander.gsicomp.on.ca (Postfix, from userid 2000) id C187E43B31; Mon, 1 Feb 2021 18:42:08 -0500 (EST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on salamander.gsicomp.on.ca X-Spam-Level: X-Spam-Status: No, score=-2.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=unavailable autolearn_force=no version=3.4.2 Received: from HEXEN (dhcp-108-168-19-149.cable.user.start.ca [108.168.19.149]) by mail.salamander.gsicomp.on.ca (Postfix) with ESMTPA id DD3C043B2F; Mon, 1 Feb 2021 18:42:07 -0500 (EST) From: "Matt Emmerton" To: "'Polytropon'" Cc: References: <012a01d6f81e$3103d390$930b7ab0$@gsicomp.on.ca> <20210201155842.1e529018.freebsd@edvax.de> In-Reply-To: <20210201155842.1e529018.freebsd@edvax.de> Subject: RE: Help recovering damaged drive - fsck segfaults, read-only mount looks ok Date: Mon, 1 Feb 2021 18:42:04 -0500 Message-ID: <016b01d6f8f3$d8853c00$898fb400$@gsicomp.on.ca> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Content-Language: en-ca Thread-Index: AQJ7oIlz2kGkp76vUMQE2AuCGU6OzgJwo/poqOcf5dA= X-Rspamd-Queue-Id: 4DV4H86MQvz3PkB X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gsicomp.on.ca header.s=duo-1597798936401-95ae6877 header.b=yrnTo/RP; dkim=pass header.d=outbound.mailhop.org header.s=dkim-high header.b=u/ZhR3x+; arc=pass (outbound.mailhop.org:s=arc-outbound20181012:i=1); dmarc=none; spf=pass (mx1.freebsd.org: domain of matt@gsicomp.on.ca designates 54.213.22.21 as permitted sender) smtp.mailfrom=matt@gsicomp.on.ca X-Spamd-Result: default: False [-4.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_XOIP(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:54.213.22.21]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[gsicomp.on.ca:+,outbound.mailhop.org:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[54.213.22.21:from]; ASN(0.00)[asn:16509, ipnet:54.213.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; ARC_ALLOW(-1.00)[outbound.mailhop.org:s=arc-outbound20181012:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gsicomp.on.ca:s=duo-1597798936401-95ae6877,outbound.mailhop.org:s=dkim-high]; FREEFALL_USER(0.00)[matt]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[gsicomp.on.ca]; SPAMHAUS_ZRD(0.00)[54.213.22.21:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[54.213.22.21:from]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Feb 2021 23:42:13 -0000 On Sun, 31 Jan 2021 17:12:40 -0500, Matt Emmerton wrote: > > If I force a mount in readonly mode, I can inspect the drive and at > > first glance, everything seems valid. Since this machine is used for > > backups, I have lots of other medata (eg, checksums) and I'm slowly > > working through to see if anything important is damaged. > > At this point: STOP. > If your data is important to you, get a copy of it NOW. Yep, I've been down this road before, which is why I have backups in the first place. Still working through which is the "best" place to seed my new backups from - the backup system (maybe damaged) or the source system (still healthy). > > From some of the stuff that fsck is finding, it's clear that the > > corruption is in a rather large-and-deep directory tree that was recently deleted. > > It's possible that the 'rm -rf' for this was running in the background > > when the system lost power. > > Therefore deleted files (or "scheduled for deletion") can still be present in the r/o > mount. The thing is - I don't care about the deleted files. What I care about is the fact that trying to "fix" the partially deleted directory tree is sending fsck down paths which might not be right, which has me worried. > > Is there any way to have fsck be more "selective" in what it checks/repairs? > > It's been a long time since I've done low-level filesystem surgery, > > but it seems to me that if I can prevent it from going off into the > > weeds (and trying to repair inode entries that are no longer > > relevant), all will be well. > > Yes. There is a "preen mode" (fsck -p) and a forced mode (fsck -f). > Be careful with specifying -y, it does not always to what you want it to do. Data loss might happen. > > See "man fsck" for details. Preen mode doesn't do anything useful (basically exits immediately). Force mode sounds scary. And I agree that fsck -y might not do the right thing. > > Any advice? I have thought about doing some inspection with "ls -i" > > and then being very selective in the inodes I get fsck to repair, but > > that seems challenging to get right. > > And _that_ is how I finally got my files back (the initial "severe data > loss problem more than 10 years ago): With ls -i, I determined the > inode of an offending directory, then used fsdb (which I found out > about reading a reference manual about a GDR UNIX system) to > remove it, and _then_ (!) fsck was able, after two runs, to bring > the filesystem back to a consistent state. Ahah! fsdb! That's the tool I remember poking around with a long time ago. > All the best, and I hope you can solve that problem. It's one of the very > few cases that can happen, and which teach you a lot about how the UFS > filesystem works. :-) Thank you for your guidance. Hopefully I can get to the bottom of this -- safely :) Matt