From owner-freebsd-questions@freebsd.org Fri Oct 1 19:53:23 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 687566B4C2A for ; Fri, 1 Oct 2021 19:53:23 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "holgerdanske.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HLglQ2h8qz3DNv for ; Fri, 1 Oct 2021 19:53:22 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1633118000; bh=KrB8QJpzq+zarTD8iQbE+fq4wAwQA1VCxFakE60S2OI=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject: Content-Language:To:References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=Mp0l/Ys8bOfD+88kUcGiWC4ST/07I6MCDHw7uJlUY6gqaacXl4gorfMY83bEoRtnk TJaEhXlB9/lYMiSRmgqbsgpMQHEuCAfcm6vpJL/ZmjU6L1Rp2p9c7p1IudnMRy2W6L dNDJh04lInFUeB2Zg8aUvjL1aZb6oIDNjyVQrWkcB/YsuhmdLzDb6TSd4dCXaE862T 6ZS5aF/qGZ+heRFRsyBCnsBKk4WohWZvs2jkVZPbAfvU80OCNi9bsGQB8h7q8dYbQd 7xTbCtpDZCeB6Ywe1Jm1so/EwGPCSEpU55p/4jPzDuBEwa8IFHPbziq97T3mpv+0gl FyY3cxK0LSsrliWdZN/LL0ZkHYvG4vFtqsWKnk0U3yHCoguwaTIupGmNj81SGPhYPw tzC/GuFPazZSbtlyLc3RROCqoNWt45ny1MKNy1Y55R3E/nsMmPyRjyAEAL4eXWDIFB mRB99F8wa/mIQab+VWvlCYviQcQ5dJqCCG/kZ07JGNtwfgRc9ocLA6imTiOaMAO/1t RhEz1/GghWcfcHc/23hkbLbd30YybCC7tgWRy2GxQ+WHTI1SPx9Cho9QmKVoTe5Dt9 QeyErjLjievZzhhrH1PypK7qYvTCNN0CPt5CL23lPz20dFqlttZ1Vu1xtbUKlAY5zD Ey0C9jaKWmeqmfZ0GVs1Epqg= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Fri, 1 Oct 2021 12:53:20 -0700 Message-ID: <87e5fe55-4b93-694e-397b-e383926fb6c2@holgerdanske.com> Date: Fri, 1 Oct 2021 12:53:11 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.1.2 Subject: Re: zfs q regarding backup strategy Content-Language: en-US To: freebsd-questions@freebsd.org References: From: David Christensen In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4HLglQ2h8qz3DNv X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b="Mp0l/Ys8"; dmarc=pass (policy=none) header.from=holgerdanske.com; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 184.105.128.27 as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[holgerdanske.com:+]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Oct 2021 19:53:23 -0000 On 10/1/21 07:14, tech-lists wrote: > Hello, > > On Thu, Sep 30, 2021 at 08:06:28PM -0700, David Christensen wrote: > >> Michael Warren Lucas is my go-to author for FreeBSD and ZFS: > > I think those were written quite a while ago? I need something covering > both the old zfs and OpenZFS. I'm looking for the differences, any > incompatibilities that might apply in my context. It'd be great if he > has more recent stuff. I find MWL's writings to be recent enough. He updates his books. I expect he has a process for updating blog posts. >> I use zfs-auto-snapshot for snapshot creation/ management, and homebrew >> commands/ scripts for replication.  The homebrew stuff is problematic. > > yeah homebrew I'm trying to avoid for lots of reasons. > >> I need to evaluate something like zfs-replicate (available as a FreeBSD >> package of the same name): > > Thanks for the suggestion. Problem with zfs-replicate is its dependence > on python, which from what I've observed historically, is a bit of a mess. > On my desktop there's lots of ports still needed that are dependent on > python2 even though upstream have retired it. They show up in > reports as EoL'd due to python2; there's loads right now and the desktop > builds its ports in poudriere weekly. Thankfully, I can find what I need in packages; and zfs-replicate is a current package. > What I'm looking for is something (ideally) written in sh. Or even perl. Available in the links I already posted. > But even before that, need to understand incremental backups properly. I learned by beating my head against the console. :-/ > Basically the backup needs to be (effectively) write-only. By that, I > mean only added to. ZFS snapshots are read-only by design. When you replicate a read-write filesystem, you replicate via snapshots. The snapshots arrive read-only, and the filesystem arrives read-write. If you touch the destination filesystem, the next incremental replication will fail. The '-F' option to 'zfs receive' is required to discard changes on the destination. I do not see an option for 'zfs receive' to set properties during replication, but you could write a script to set 'readonly=on' on the destination after replication. If so, I don't know what happens on the next incremental replication -- properties are overwritten? Replication fails? Testing is required. Eventually, the destination will fill up. Choices include replacing the media and deleting snapshots. I do the latter (by hand). > Maybe zfs snapshots aren't the way to go; Snapshots and replication are killer features of ZFS, and primary reasons why I use ZFS. > maybe rsync in the backup > direction only (apart from when something needs to be restored) would > suit my context better. I use rsync(1) to backup from non-ZFS sources to ZFS. > What do you think? "More than enough rope to shoot yourself in the foot". David