From owner-freebsd-questions@freebsd.org Fri Oct 1 03:06:38 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 875826762E3 for ; Fri, 1 Oct 2021 03:06:38 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "xray.he.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HLFPm6Xfwz3HCX for ; Fri, 1 Oct 2021 03:06:36 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1633057588; bh=Pqool6n08TZ7HtRTWMgb+bTGPz2dGTS87KhWfJCad94=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject: Content-Language:To:References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=1NZ5h9zG68x/oHy9lPZqDVsA6dxDMJP2oVUxBw5+wDKVb+B7S9U07jw6N3hIYQ5Zq vCy+Dwa+P/O3J67F6DWJIio/Ha3LYbRAPY/YNIFtiNKOa+VjutWbhDvcNuB1xODRT7 I6oYgPIYjDr3XZT4/lWrCBsMKxi09sO0hDEpj0unyqeZ58uoLuriFZ7zcjWcu4j3oW 4i9bEDrYz2cxljQAIgERm9NQl8O5ESL8lUEMuORQmpNWAo0gfL6kNPn6EUl+uFO0qZ ODRcxwpozulM7xLpvfheMZoFqQcFc145LeeVp+AAZ+826N7wFsOEF7O8GmMV/mTcp4 +/GEWYyEiLP4x057ZU6WmMxqo3q5RjGLyKF78Uqgu7YXvq5EJwveOzPxpHBZgCYhcO c2V9Vx5uZ+XGMAT8cBRAwtDRXCJs1ZLMt0cGfb5AAD2VHbytWI3SUkdZU/nMtL8QMz F+d9CZRdZ8OHHxhwfU/HiBxUcos7k5mU5jNxdv49pQs0FZ36aP8ffAywzY0JprYbU1 gM0lU6nGgt14cvz7kGO4fFqhdAMqhMheWTJuWRZ71YYE0WOoWFoXc1V5eo+jNNcjdS ky+AZyeKVU2/6kNDqrG7O7fC4u36pwUaiouzv2YUMzkUckBs6VOZ3zy3vR3tuYcvcR hHc0VobPURVt4SAdzPX/KeR4= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Thu, 30 Sep 2021 20:06:28 -0700 Message-ID: Date: Thu, 30 Sep 2021 20:06:28 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.1.2 Subject: Re: zfs q regarding backup strategy Content-Language: en-US To: freebsd-questions@freebsd.org References: From: David Christensen In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4HLFPm6Xfwz3HCX X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b=1NZ5h9zG; dmarc=pass (policy=none) header.from=holgerdanske.com; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 184.105.128.27 as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DKIM_TRACE(0.00)[holgerdanske.com:+]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-questions]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Oct 2021 03:06:38 -0000 On 9/30/21 16:48, tech-lists wrote: > Hi, > > This is a bit chicken-and-egg for me, in that I might be formulating the > wrong question, and so getting wrong answers, so I apologise in advance > if I'm being dense. > > Basically the end result I want is to be in a position at a backup site > with full backups that can be restored to the live site if needed, or > even to get just one required file from that backup, given the filename > and modification time. The backup site has very much more diskspace than > the live site. > > I know how to make the full backup and to send it. What I'm looking for > is incremental (or would it be differential) zfs backups (snapshots). So > let's say an incremental/differential backup to the remote site happens > every 6hrs. > > If 10 incremential backups happen after the full one, and on the > live service someone loses a few files, later adding another few, maybe > a couple of those have the same name as before just with a different > modification time, all the while this is happening 24hrs pass. > > How are the accumulated differential/incremental snaps incorporated into > the main backup? Would I need to run something else on the backup > server to integrate the snaps with the full backup? Or would all the > snaps need to be kept? > How would one tell if the required files in this example are in one snap > and not in the others? > > If the policy was to keep everything in backup regardless of whether it > was deleted or not from the live service (so the backup is like a > write-only representation) I guess the approach would be different to > say if the backup just had to be the latest representation of the data > on the live server. But if it's "keep everything" then what happens > if the live server makes foo.txt then deletes it then makes foo.txt 10 > days later in the same place? All the while these > (incremental/differential) snaps have been arriving every 6 hrs. > > Additionally the live site is the old zfs on releng/12.2-p10 and the > backup site is with the OpenZFS on stable/13 which might be an issue > if they handle things significantly differently. > > thanks advance for any suggestions or any scripts/tools you can > recommend. I have a SOHO environment with a FreeBSD 12.2-RELEASE machine providing Samba and CVS services. Michael Warren Lucas is my go-to author for FreeBSD and ZFS: https://mwl.io/nonfiction/os https://mwl.io/archives/2128 https://mwl.io/archives/2140 I use zfs-auto-snapshot for snapshot creation/ management, and homebrew commands/ scripts for replication. The homebrew stuff is problematic. I need to evaluate something like zfs-replicate (available as a FreeBSD package of the same name): https://pypi.org/project/zfs-replicate/ David