From nobody Mon Nov 28 10:15:09 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NLLvD0wN3z4jkf0 for ; Mon, 28 Nov 2022 10:15:20 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4NLLvB5ynsz3DWd for ; Mon, 28 Nov 2022 10:15:18 +0000 (UTC) (envelope-from marck@rinet.ru) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of marck@rinet.ru designates 195.54.192.68 as permitted sender) smtp.mailfrom=marck@rinet.ru; dmarc=pass (policy=none) header.from=rinet.ru Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.5/8.14.5) with ESMTP id 2ASAF9h9022809; Mon, 28 Nov 2022 13:15:09 +0300 (MSK) (envelope-from marck@rinet.ru) Date: Mon, 28 Nov 2022 13:15:09 +0300 (MSK) From: Dmitry Morozovsky To: andy thomas cc: Freddie Cash , Bob Friesenhahn , Mark Saad , FreeBSD Filesystems Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-ID: 6B691B03 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (woozle.rinet.ru [0.0.0.0]); Mon, 28 Nov 2022 13:15:10 +0300 (MSK) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.70)[-0.701]; DMARC_POLICY_ALLOW(-0.50)[rinet.ru,none]; R_SPF_ALLOW(-0.20)[+ip4:195.54.192.68]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:8331, ipnet:195.54.192.0/19, country:RU]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,simple.dallas.tx.us,longcount.org,FreeBSD.org]; FREEFALL_USER(0.00)[marck]; RCPT_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4NLLvB5ynsz3DWd X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Thu, 17 Nov 2022, andy thomas wrote: > On Thu, 17 Nov 2022, Freddie Cash wrote: > > > Now that you have it working with rsync, you should look into using ZFS > > send/recv as an alternative. You should find it finishes a lot quicker than > > rsync, although it does require a bit more scripting know-how (especially if > > you want to use restartable/interruptible transfers, or use a transport > > other than SSH for better throughout). > > ZFS send/recv works "below" the filesystem later today rsync works at. ZFS > > knows which individual blocks on disk have changed between snapshots and > > only transfers those blocks. There's no file comparisons and hash > > computations to work out between the hosts. > > > > Transferring the initial snapshot takes a long time, though, as it has to > > transfer the entire filesystem across. Transferring individual snapshots > > after that takes very little time. It's similar to doing a "full" backup, > > and then "incrementals". > > > > When transferring data between ZFS pools with similar filesystem > > hierarchies, you really should consider send/recv. > > Point taken! Three days ago, one of our HPC users who has ~9TB of data stored > on our server decided to rename a subdirectory containing ~4TB of experimental > data stored as many millions of relatively small files within a lot of > subdirectories. As a result, rsync on the destination (mirror) server is still > deleting his old folder and its contents and hasn't even started mirroring the > renamed folder. > > Since our servers have been up for 5.5 years and are both well overdue for an > O/S upgrade from FBSD 11.3 to 13.x anyway, I think this would be a good > opportunity to switch from rsync to ZFS send/recv. I was planning to do the > O/S update over the upcoming Christmas vacation when HPC demand here > traditionally falls to a very low level - I will set up a pair of test servers > in the next day or two, play around with this and get some experience of this > before upgrading the 'live' servers. [snip] you may look at zxfer port as a start point of ZFS send/recv scripting excerpt from our backup script: SSHOPT="-i /home/backup/.ssh/id_ed25519 backup@$MACHINE" zxfer -FkPv -U -o compression=lz4 \ -O "$SSHOPT" \ -R $POOL ${BASEFS}/zfs/$MACHINE (initiated from backup machine to pull snapshots missing for the moment; target servers use zfsnap2 port daily with 6d/5w/3m retention for d/w/m snapshots respectively) -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] --------------------------------------------------------------------------- *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- woozle@woozle.net *** ---------------------------------------------------------------------------