From nobody Fri Sep 23 22:09:25 2022 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MZ5sv5B50z4d3yR for ; Fri, 23 Sep 2022 22:09:39 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MZ5st3Zw5z3gV2 for ; Fri, 23 Sep 2022 22:09:38 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1663970970; bh=R0qq6OZw9K1ASj8MJH+ONrThK+n8aQfX6Q/uccn09jw=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject: Content-Language:To:References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=1d+StAuxBsDF29QsAc1YAY0JTROzA13ZPIVLjehGspCDESZOYIKNZKB2s2XzzxtwK Vhd2jzpv8YP4CiyjFUmsxNxj2Iu/GqKA0AjMVqwNyDAeCwAR+8P6yLc2zgR+7JFlT/ rH88t/vvUrgIs/joAGLS2SukUOjenPi3Tc9hkQFh7oZ1VdOy0oV6GmLgaVUFKBNE47 v+HbLp+U5h4cYPHEwCZB2CaHJcdfZGuTsU/mUtFlB8hL/N1vjoDsdQTpH6ngp6+fSO Rlx8N95A6TbOTgj5cXAXIG4dGaDTy+UdtXIzYGdzh+A5q+7XrkBa9apz0Y1dKbNE+O 5BrkS4hTksJZCejms8cdZWjBQ4B6VgprGlt/daLkuXc8Ulam4ylURr9S9C0kHcjQ4n 89jcqjIdv8eDzlR9rQCdsPrdGlA+xk7uirpPdI8gv2wDMiZI4CtOFoWTJ4F6kSNEgT I2KnP/RzuSHKgGY0Hr7mo3X4N1de5AFUDXY8+LQNuLBznYSmKSHrttRlZxNJkybXtu Ez20EmpaH0EqnnBMDCdhqI2zHTWxk0h8ZZbgnKpLOcrkOzb8VlosTJ1mk0qtABkFHo fghx61vyuvSCjDHhDfUjCDxJFWEZhkG/KEi6MmKfdlbHbpmdkN10Bjh9BHkpFoRcvs IxE2dm0vhXM2KBajk7w5Wmg8= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Fri, 23 Sep 2022 15:09:30 -0700 Message-ID: Date: Fri, 23 Sep 2022 15:09:25 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.12.0 Subject: Re: zfs replication tool Content-Language: en-US To: questions@freebsd.org References: <20220920092905.3k7qzt7lvhywhcfn@x1> <20220920122029.ufsoyo47qnxtmcqk@x1> <20220920132503.8bfd124b9697d90e474086b3@sohara.org> <20220920125154.n7lcqr3xzhsrylzr@x1> <20220920142314.df043149e82d63cbd289a92e@sohara.org> <4cf58433-a13b-9b35-fc3e-8fad6b2ac93d@sentex.net> <20220920141006.n6x6mxpafy3zqpk3@x1> <20220923080305.alrgrrnxljtweo3i@x1> <498d478e-7fe7-1840-6263-930b498c5bbf@holgerdanske.com> <20220923092535.jtrqgcxgxq3sstgp@x1> From: David Christensen In-Reply-To: <20220923092535.jtrqgcxgxq3sstgp@x1> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4MZ5st3Zw5z3gV2 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b=1d+StAux; dmarc=pass (policy=none) header.from=holgerdanske.com; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 184.105.128.27 as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; R_SPF_ALLOW(-0.20)[+a]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[holgerdanske.com:+]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; MLMMJ_DEST(0.00)[questions@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 9/23/22 02:25, Julien Cigar wrote: > On Fri, Sep 23, 2022 at 01:57:54AM -0700, David Christensen wrote: >> On 9/23/22 01:03, Julien Cigar wrote: >> >>> I'm still getting replication errors with zrepl on my test installation, >>> see https://github.com/zrepl/zrepl/issues/631 for details. The initial >>> transfer works, but after that I'm getting weird "cannot receive >>> incremental stream: destination ..." messages, despite all snapshots >>> have been well preserved on both side >> Does zrepl have xtrace, verbose, debug, etc., options? >> > > yes, but I haven't found an explanation why some snapshots aren't > transferred > >> >> I wrote homebrew scripts to automate ZFS replication and I saw that error >> message many times in the past. The solution was to add the '-F' option to >> the 'zfs receive' commands. Is there a way to do this with zrepl? >> > > I'm not sure, and even if there was the possibility I'd like to > understand why it is needed as all snapshots are there. Rollback to the > most recent snapshot shouldn't be necessary in this case.. On 9/23/22 02:52, Julien Cigar wrote: > mmh could it be because target dataset is mounted and because of atime > (or ...) issues...? AIUI FreeBSD < 13 is ZOL and 13 <= FreeBSD is OpenZFS. The difference may matter. Please run and post: $ freebsd-version ; uname -a For my backup via ZFS replication process, I used to automatically import the destination pool read-write on the backup server at boot time via /etc/rc.conf, and I used to see the "cannot receive incremental stream: destination pool/dataset has been modified" error message. Now I run a script to import the destination pool using the -R (altroot) and '-o ro' (mount filesystems readonly) options. I am considering trying the -N (do not mount filesystems) option instead of '-o ro'. Now I typically see the error message only when I have modified the topology and/or snapshots of the live or backup datasets (e.g. the cause of the error message is known). If you see an unexpected error message again, please fully document it and post the documentation somewhere (redact confidential information as needed): 1. Listing of snapshots of involved datasets; source and destination. 2. Properties of involved pools and dataset(s); source and destination. 3. zrepl command, arguments, options, configuration, standard input, standard output, standard error, logs, reports, whatever, with xtrace, versbose, debug, etc., enabled. On 9/23/22 04:24, Julien Cigar wrote: > it looks like there is no way to specify a -F on the receive side with > zrepl.. my problem looks like > https://github.com/zrepl/zrepl/issues/408 issue. I am unsure if a lack of '-F' is a bug or a feature. zrepl is far more sophisticated than anything I do; their model might eliminate the need for '-F'. On 9/23/22 05:43, Julien Cigar wrote: > setting mountpoint=none on the target dataset fixed the issue. > Apparently mounting a dataset, even as read-only, somewhat "alters" it > and incremental snapshot replication fails afterwards. > > So I'm setting mountpoint=none That sounds like a variation of the '-o ro' and '-N' themes. '-o ro' has the advantage that you can see the endpoint filesystems and volumes; for recovery, validation, whatever. I assume '-N' mounts neither the filesystem nor the snapshots, but mounting just the snapshots could be useful. '-o mountpoint=none' would seem to have similar practical effect as '-N'. > and switched to clones and promotes I played with clones years ago. I do not recall trying 'promote'. Fortunately, I do not need clones. David