From owner-freebsd-questions@freebsd.org Sat Oct 2 23:21:34 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 446DE6AC200 for ; Sat, 2 Oct 2021 23:21:34 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HMNK92843z3N60 for ; Sat, 2 Oct 2021 23:21:33 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id ABBBA5C00D5 for ; Sat, 2 Oct 2021 19:21:27 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Sat, 02 Oct 2021 19:21:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm3; bh=er5kDnT1sjCaalRv/AKZHCXA8ZX 19egm/sLvRhJmMmQ=; b=kOmCk0DAFquUwPvKBVZbHotROJuZM/61EkoKpx0blTx 1VKmKGZ6pK3Lg7j939k1xejex6yhkMfj/CV8Q2zjt8JxoicNv47zZw563kCLAD+Y 9iY+m4ZLwtEaUIKvyS9pICUqK8aY4LEV9daL72ZxHlRcONya11KmCiZjT9UdaUUI R9x2u2X6vu5Zb6hCdMyoYRmKdp3/WPD2ytdRGSgdZ8CHQ1LdPaIROF6UrmCiwaLL iYt2DOnhYTyq5crMAS2cjZPZV0E99NsBFcHS6oahYbPHa+2t6MjZSYg+Dy3gyNs3 FzgLzXWSrVmtXX/pICUfVvZ95QwH6An6Gs2pccy2sVQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=er5kDn T1sjCaalRv/AKZHCXA8ZX19egm/sLvRhJmMmQ=; b=XW32Rujb3g2ZYebae6+OQ8 rxQFFWLNob2L8UB5vWF4Vf6T44Z2nATNJWiAVMJ6tfk1Y7rmIoiv8j6d/7NgFXjl G3iN/EoWHNmZMtWyfqd96yuSm0lgC/XRG5zk6zeIVMtOOMRHqE8GthdjnBbLR0Lz /kiKDOSicOtuFrIveyhTJqZJ+ht84F3QpZvYrNUsV6ONWxtC6bPV3Rfxe8Ub3cNv iaxV41IlWwm7pRKndacvfqK7uDv0xZ7XXPmND5Zjo3qhbL/2LcMHb1vGx3OLYdOC o70EMjyEdpi/EfdtcpuH8CTA9rD/XiZtqGwpQUIxt/Tx1wSJ0DxrdneaC4L6glxw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrudekledgvdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseii hiigshhtrdhnvghtqeenucggtffrrghtthgvrhhnpedtheeigfdvudefkeekvddtfedvte dttdekuddvgeevlefftdekffdujedvhfduteenucevlhhushhtvghrufhiiigvpedtnecu rfgrrhgrmhepmhgrihhlfhhrohhmpehtvggthhdqlhhishhtshesiiihgihsthdrnhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sat, 2 Oct 2021 19:21:27 -0400 (EDT) Date: Sun, 3 Oct 2021 00:21:24 +0100 From: tech-lists To: freebsd-questions@freebsd.org Subject: Re: zfs q regarding backup strategy Message-ID: Mail-Followup-To: freebsd-questions@freebsd.org References: <20211001222816.a36e9acbd4e8829aed3afb68@sohara.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Cvx1CJLqaQgzmSfz" Content-Disposition: inline In-Reply-To: <20211001222816.a36e9acbd4e8829aed3afb68@sohara.org> X-Rspamd-Queue-Id: 4HMNK92843z3N60 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm3 header.b=kOmCk0DA; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=XW32Rujb; dmarc=none; spf=none (mx1.freebsd.org: domain of tech-lists@zyxst.net has no SPF policy when checking 66.111.4.28) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-6.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[66.111.4.28:from]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.0.0/20, country:US]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm3,messagingengine.com:s=fm1]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; DMARC_NA(0.00)[zyxst.net]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; R_SPF_NA(0.00)[no SPF record]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Oct 2021 23:21:34 -0000 --Cvx1CJLqaQgzmSfz Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 01, 2021 at 10:28:16PM +0100, Steve O'Hara-Smith wrote: >On Fri, 1 Oct 2021 15:14:08 +0100 >tech-lists wrote: > >> What I'm looking for is something (ideally) written in sh. Or even perl. >> But even before that, need to understand incremental backups properly. >> Basically the backup needs to be (effectively) write-only. By that, I >> mean only added to. > > With zfs send every stream is a send up to some 'target' snapshot, >a full stream is the complete history while an incremental stream is from >an earlier 'source' snapshot. > > It is therefore necessary for any successful incremental backup >system to know what the last snapshot sent was so that the next cycle can >start by creating a new snapshot and sending the increment from the last >sent to the latest. On the receiving end the options are to either store >each send as a file named in such a way as to make it obvious what order to >process them in should restore be needed or (more usefully) to use zfs recv >to load them into filesystems in a pool on the receiving machine. If you do >this there will be no pruning of the received snapshots unless you set >something up to do it. OK. if I think I understand correctly then if we need to keep everything th= en=20 I'll need to keep every incremental snapshot. Let's say there are 3 files in a zfs dir 1, 2 & 3 full backup happens on day 1, all three files are fully backed up first snapshot happens on day 2: theres nothing added nothing removed becau= se nothing changed at source second snapshot happens on day 3, file 2 disappears at source so the=20 difference is this file is not there but the other two are. third snapshot happens file 2 of the same name appears at source but has different content. A month (30 days) later someone looks for file 2 as it was on day 1, and it's not the one they expected on the source. So I have to get it from the backup somehow. In order to do this, and given that when I'm looking for this I might not know *when* file 2 was deleted from the source, then changed and then added, do I have to keep every single snapshot? If this is the case, then how can I scan through the snapshots looking for the right file? there may be thousands of snapshots. Or can I incorporate or expire the snapshots into the full backup and tell it that I want to look back to day 1 and retreive file 2? The reason I'm thinking this might be possible is via zfs redundancy. copies=3D3 that sort of thing. Like a time machine but for a backup, I guess. Or some other method that I've not understood (there's lots about zfs I don't understand) or haven't read about yet? thanks, --=20 J. --Cvx1CJLqaQgzmSfz Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAmFY6WsACgkQs8o7QhFz NAVELw//bi/KHoGaBOLH+cxTBlZJsuQ7SYNwdQk+nWGvU/sOS21yPJCAgaCfg5PU cc4r6SCxnYtlEY9hmmhipqJgqWUES19yKoYWlgVIropYRXeO/iQpvixzY7HlI3uN XX0gVBNh3i0Q3CC2FPNBgsRPOfWgLskkDmw+cRSSRbp8DZJU8Vq6R3Rs4QoNQpC7 y34yTtK1cnNNIMdI52ZuJ1++28lBN+qR7wMam5vIdT6DEKfzaAPKn6FwuVSOMTZG v3eTipCocM1jpHPuA2PrVF9tg/Wc4trORKbRvnqpWpm+2/4WJr7vuHqCfRQD70o9 byptIb3SVU7OhHB/XCHSyV2MyDSjpFXsOJD/CDDXUgG6sNCtQ4+hqF6+1ZWF4EpM ORmvqjrF+peOrLbS0QFA7f+XKNVPqvOwR+yT75WFLrUs48hzeOMMzAsUwFgFaN3z /IertI7zl3wAjJ0GTg545PgByPZXR83G+IVv2JbD1mGZi6vxYpugLoDqlqHt9tOO 7DqHgHXzrxkbkOP4RnO30mVdEa2f5HPCW/NsbnOHvxyzB4YFGBTc0/PIFlGN81hW +PJrrmnjAaQKYdQEPdEZGRGFTiaFdV0MtY/7fJs0LBJhIprntstmWCztoPHDdh0W uaCHbQoQfouC0HkKK8GgOioZUdSqpeWcbxb3tCRDya3xbdhr9XE= =7zsW -----END PGP SIGNATURE----- --Cvx1CJLqaQgzmSfz--