Date: Mon, 28 Nov 2022 13:15:09 +0300 (MSK) From: Dmitry Morozovsky <marck@rinet.ru> To: andy thomas <andy@time-domain.co.uk> Cc: Freddie Cash <fjwcash@gmail.com>, Bob Friesenhahn <bfriesen@simple.dallas.tx.us>, Mark Saad <nonesuch@longcount.org>, FreeBSD Filesystems <freebsd-fs@FreeBSD.org> Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync Message-ID: <alpine.BSF.2.00.2211281310250.349@woozle.rinet.ru> In-Reply-To: <alpine.BSF.2.22.395.2211171422520.50255@mail0.time-domain.net> References: <alpine.BSF.2.22.395.2211111709230.29479@mail0.time-domain.net> <CAOgwaMuoLQ9Er67Y%2B=q%2Bf9724v2WT3L5v5TZaRVXq%2B=1vEyJ%2BA@mail.gmail.com> <alpine.BSF.2.22.395.2211112008220.30520@mail0.time-domain.net> <alpine.GSO.2.20.2211121949060.7126@scrappy.simplesystems.org> <CAMXt9Nbr=7K6PELVGAPZ=-RiAfx=zp9iOoKyWdH=0H2=AiE52Q@mail.gmail.com> <alpine.GSO.2.20.2211131137020.7126@scrappy.simplesystems.org> <alpine.BSF.2.22.395.2211170840040.46246@mail0.time-domain.net> <CAOjFWZ6vxeXonEDndUvLkudRsRsBAd0sJ5ssOf4gLCgkVgSeyQ@mail.gmail.com> <alpine.BSF.2.22.395.2211171422520.50255@mail0.time-domain.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 17 Nov 2022, andy thomas wrote:
> On Thu, 17 Nov 2022, Freddie Cash wrote:
>
> > Now that you have it working with rsync, you should look into using ZFS
> > send/recv as an alternative. You should find it finishes a lot quicker than
> > rsync, although it does require a bit more scripting know-how (especially if
> > you want to use restartable/interruptible transfers, or use a transport
> > other than SSH for better throughout).
> > ZFS send/recv works "below" the filesystem later today rsync works at. ZFS
> > knows which individual blocks on disk have changed between snapshots and
> > only transfers those blocks. There's no file comparisons and hash
> > computations to work out between the hosts.
> >
> > Transferring the initial snapshot takes a long time, though, as it has to
> > transfer the entire filesystem across. Transferring individual snapshots
> > after that takes very little time. It's similar to doing a "full" backup,
> > and then "incrementals".
> >
> > When transferring data between ZFS pools with similar filesystem
> > hierarchies, you really should consider send/recv.
>
> Point taken! Three days ago, one of our HPC users who has ~9TB of data stored
> on our server decided to rename a subdirectory containing ~4TB of experimental
> data stored as many millions of relatively small files within a lot of
> subdirectories. As a result, rsync on the destination (mirror) server is still
> deleting his old folder and its contents and hasn't even started mirroring the
> renamed folder.
>
> Since our servers have been up for 5.5 years and are both well overdue for an
> O/S upgrade from FBSD 11.3 to 13.x anyway, I think this would be a good
> opportunity to switch from rsync to ZFS send/recv. I was planning to do the
> O/S update over the upcoming Christmas vacation when HPC demand here
> traditionally falls to a very low level - I will set up a pair of test servers
> in the next day or two, play around with this and get some experience of this
> before upgrading the 'live' servers.
[snip]
you may look at zxfer port as a start point of ZFS send/recv scripting
excerpt from our backup script:
SSHOPT="-i /home/backup/.ssh/id_ed25519 backup@$MACHINE"
zxfer -FkPv -U -o compression=lz4 \
-O "$SSHOPT" \
-R $POOL ${BASEFS}/zfs/$MACHINE
(initiated from backup machine to pull snapshots missing for the moment;
target servers use zfsnap2 port daily with 6d/5w/3m retention for d/w/m
snapshots respectively)
--
Sincerely,
D.Marck [DM5020, MCK-RIPE, DM3-RIPN]
[ FreeBSD committer: marck@FreeBSD.org ]
---------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- woozle@woozle.net ***
---------------------------------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.2211281310250.349>