From nobody Fri Sep 23 22:42:19 2022 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MZ6bj2Kxzz4d75G for ; Fri, 23 Sep 2022 22:42:25 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MZ6bh32yhz3lh3 for ; Fri, 23 Sep 2022 22:42:24 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1663972939; bh=4PEZ5Zth+Wt+fRzcwLpXjHPuUIWfiKx+jMNkQcf9cH8=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject: Content-Language:To:References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=umiu38jYP8n5HrtoYgp4gHbmMs3j+D85IDUZK+tfb+6cBtJbbw/x4u9u11c7UVP/r JZsD265Nz/U9S5dm2y3mDQEN/gwbLCA/1NeI+cqjGR0t4Pgr89eIsmBa/zwvQUvS0W geG0puSIY9UMjE8PlLdeSvey4Xv+NSYH6Nvm6IBR7cj2ZUikoSlzi8mRMJixpFP9PL MPXVaXULJkL+QZaLRGIGISOOT9ogw18tZ2q5yyqNYNu3Hf3ntyeLvjrPmGGn0eXzEa LxD0buoH9IhBKMzhOU7c2SHxjsIxmmaqDyHzMgxQ5jIfgtwrS5c2dkldi7lDeVDdLk uBQN4fAU5UzESnmGrt5oKNrN+FRuPjTX/wdXC1tfACwJVwp84XeV/QHz69G6VXgYld biIUdUeq4VxRRbSKogaYwIDx9AEBkoF4VadaGKmGIf8Bsusom5B6EG6WYh5khUQPBa AJBO2FsWsfmweNDVp45ab3zzijVdqmYzmonxh30h8w9BWrDXt2r8LvS5QRinh/Kg2F XD4a7L42rrYuPEmlvi1bq0Ezm6vZxMuIyaYZvEW9JLEWuaRbBcQcAdjJoqdQu0WKJD jYcyQCoR2YJCTSiiglFN4JQ/MU321mpFaeiKCwnV/qZ6ocPdNSieWqfBi10wVABvK+ bYAmsClMy/isKR0MZxhx5lFc= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Fri, 23 Sep 2022 15:42:19 -0700 Message-ID: <7be8f47b-bfea-ef5a-7b59-2f94f8d310e2@holgerdanske.com> Date: Fri, 23 Sep 2022 15:42:19 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.12.0 Subject: Re: data, metadata, backup, and archive integrity and correction Content-Language: en-US To: questions@freebsd.org References: From: David Christensen In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4MZ6bh32yhz3lh3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b=umiu38jY; dmarc=pass (policy=none) header.from=holgerdanske.com; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 184.105.128.27 as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[holgerdanske.com:+]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; MLMMJ_DEST(0.00)[questions@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 9/23/22 07:28, Ralf Mardorf wrote: > On Wed, 2022-09-21 at 03:58 -0700, David Christensen wrote: >> Integrity checking of data and metadata is important -- both for live >> data and, especially, for backups and archives. > > Hi David, > > good to read that I'm not alone :). > > I don't care that much for live data. To loose a few minutes or even a > few days isn't fun, but bearable. Backups and archives can contain a > full life. > > Unfortunately integrity checks are way to often a PITA :(. > > My current strategy is an obsessional neurosis. It's a little bit more a > psychopathological workaround, than a computer technical attempt. > > I have got tons of backups (unfortunately not stored in different > locations, just on different drives) and from time to time I restore > backups or archives using a sandbox, just for testing purpose. Redundancy of backups is a good thing. So is taking backup media out of rotation and archiving it. "Defense in depth." "Don't put all of your eggs in one basket." One of the reasons I wanted storage with built-in integrity checking and correction is because of an experience I had using storage without those features. I believe I was migrating my bulk data from an older, smaller store to a newer, bigger store. I wrote a script to validate the move (using cmp(1)?). I checked the new store against the old store -- okay. I checked the new store against a backup of the old store -- one photograph file differed by a few bits. All versions of the photograph file opened correctly with a viewer. All photographs looked the same on the screen. But, at least one file is corrupt. Which file(s)? I never figured it out. I kept all versions of the file. (And, I have kept all camera media.) David