Date: Thu, 17 Nov 2022 01:31:02 -0800 From: Freddie Cash <fjwcash@gmail.com> To: andy thomas <andy@time-domain.co.uk> Cc: Bob Friesenhahn <bfriesen@simple.dallas.tx.us>, Mark Saad <nonesuch@longcount.org>, FreeBSD Filesystems <freebsd-fs@freebsd.org> Subject: Re: Odd behaviour of two identical ZFS servers mirroring via rsync Message-ID: <CAOjFWZ6vxeXonEDndUvLkudRsRsBAd0sJ5ssOf4gLCgkVgSeyQ@mail.gmail.com> In-Reply-To: <alpine.BSF.2.22.395.2211170840040.46246@mail0.time-domain.net> References: <alpine.BSF.2.22.395.2211111709230.29479@mail0.time-domain.net> <CAOgwaMuoLQ9Er67Y%2B=q%2Bf9724v2WT3L5v5TZaRVXq%2B=1vEyJ%2BA@mail.gmail.com> <alpine.BSF.2.22.395.2211112008220.30520@mail0.time-domain.net> <alpine.GSO.2.20.2211121949060.7126@scrappy.simplesystems.org> <CAMXt9Nbr=7K6PELVGAPZ=-RiAfx=zp9iOoKyWdH=0H2=AiE52Q@mail.gmail.com> <alpine.GSO.2.20.2211131137020.7126@scrappy.simplesystems.org> <alpine.BSF.2.22.395.2211170840040.46246@mail0.time-domain.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000011651005eda74014
Content-Type: text/plain; charset="UTF-8"
Now that you have it working with rsync, you should look into using ZFS
send/recv as an alternative. You should find it finishes a lot quicker than
rsync, although it does require a bit more scripting know-how (especially
if you want to use restartable/interruptible transfers, or use a transport
other than SSH for better throughout).
ZFS send/recv works "below" the filesystem later today rsync works at. ZFS
knows which individual blocks on disk have changed between snapshots and
only transfers those blocks. There's no file comparisons and hash
computations to work out between the hosts.
Transferring the initial snapshot takes a long time, though, as it has to
transfer the entire filesystem across. Transferring individual snapshots
after that takes very little time. It's similar to doing a "full" backup,
and then "incrementals".
When transferring data between ZFS pools with similar filesystem
hierarchies, you really should consider send/recv.
Cheers,
Freddie
Typos due to smartphone keyboard.
On Thu., Nov. 17, 2022, 12:50 a.m. andy thomas, <andy@time-domain.co.uk>
wrote:
> I thought I would report back that changed my rsync options from '-Wav
> --delete' to '-av --inplace --no-whole-file --delete' has made a
> significant difference, with mirrored directory sizes on the slave server
> now falling and approaching the original sizes on the master. The only
> downside is that since whole-file replication is obviously a lot faster
> than updating the changed parts of individual files, mirroring is now
> taking longer than 24 hours so this will be changed to every few days or
> even weekly when more is known about user behaviour on the master server.
>
> Andy
>
> On Sun, 13 Nov 2022, Bob Friesenhahn wrote:
>
> > On Sun, 13 Nov 2022, Mark Saad wrote:
> >>>
> >> Bob are you saying when the target is zfs --inplace --no-whole-file
> helps
> >> or just in general when you have
> >> large files ? Also have you tried using --delete-during /
> --delete-after
> >> ?
> >
> > The '-inplace --no-whole-file' updates the file blocks if they have
> changed
> > (comparing the orgin blocks with the existing mirror blocks) rather than
> > creating a new copy of the file and moving it into place when it is
> complete.
> > ZFS does not check if data content has been changed while it is being
> written
> > so a write of the same data will result in a fresh allocation based on
> its
> > Copy On Write ("COW") design. Writing a whole new file obviously
> > significantly increases the number of blocks which are written.
> Requesting
> > that rsync only write to the file for the blocks which have changed
> reduces
> > the total number of blocks which get written.
> >
> > The above helps quite a lot when using snapshots since then fewer blocks
> are
> > in the snapshots.
> >
> > I have never tried --delete-during so I can't comment on that.
> >
> > Bob
> > --
> > Bob Friesenhahn
> > bfriesen@simple.dallas.tx.us,
> http://www.simplesystems.org/users/bfriesen/
> > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
> > Public Key,
> http://www.simplesystems.org/users/bfriesen/public-key.txt
> >
> >
>
>
> ----------------------------
> Andy Thomas,
> Time Domain Systems
>
> Tel: +44 (0)7866 556626
> http://www.time-domain.co.uk
>
>
--00000000000011651005eda74014
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"auto">Now that you have it working with rsync, you should look =
into using ZFS send/recv as an alternative. You should find it finishes a l=
ot quicker than rsync, although it does require a bit more scripting know-h=
ow (especially if you want to use restartable/interruptible transfers, or u=
se a transport other than SSH for better throughout).<div dir=3D"auto"><br>=
</div><div dir=3D"auto">ZFS send/recv works "below" the filesyste=
m later today rsync works at. ZFS knows which individual blocks on disk hav=
e changed between snapshots and only transfers those blocks. There's no=
file comparisons and hash computations to work out between the hosts.</div=
><div dir=3D"auto"><br></div><div dir=3D"auto">Transferring the initial sna=
pshot takes a long time, though, as it has to transfer the entire filesyste=
m across. Transferring individual snapshots after that takes very little ti=
me. It's similar to doing a "full" backup, and then "inc=
rementals".</div><div dir=3D"auto"><br></div><div dir=3D"auto">When tr=
ansferring data between ZFS pools with similar filesystem hierarchies, you =
really should consider send/recv.<br><div dir=3D"auto"><br><div data-smartm=
ail=3D"gmail_signature" dir=3D"auto">Cheers,<br>Freddie<br><br>Typos due to=
smartphone keyboard.</div></div></div></div><br><div class=3D"gmail_quote"=
><div dir=3D"ltr" class=3D"gmail_attr">On Thu., Nov. 17, 2022, 12:50 a.m. a=
ndy thomas, <<a href=3D"mailto:andy@time-domain.co.uk">andy@time-domain.=
co.uk</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I thought I wo=
uld report back that changed my rsync options from '-Wav <br>
--delete' to '-av --inplace --no-whole-file --delete' has made =
a <br>
significant difference, with mirrored directory sizes on the slave server <=
br>
now falling and approaching the original sizes on the master. The only <br>
downside is that since whole-file replication is obviously a lot faster <br=
>
than updating the changed parts of individual files, mirroring is now <br>
taking longer than 24 hours so this will be changed to every few days or <b=
r>
even weekly when more is known about user behaviour on the master server.<b=
r>
<br>
Andy<br>
<br>
On Sun, 13 Nov 2022, Bob Friesenhahn wrote:<br>
<br>
> On Sun, 13 Nov 2022, Mark Saad wrote:<br>
>>> <br>
>> Bob are you saying when the target is zfs --inplace --no-whole-fil=
e helps<br>
>> or just in general when you have<br>
>> large files ?=C2=A0 Also have you tried using --delete-during / --=
delete-after <br>
>> ?<br>
><br>
> The '-inplace --no-whole-file' updates the file blocks if they=
have changed <br>
> (comparing the orgin blocks with the existing mirror blocks) rather th=
an <br>
> creating a new copy of the file and moving it into place when it is co=
mplete. <br>
> ZFS does not check if data content has been changed while it is being =
written <br>
> so a write of the same data will result in a fresh allocation based on=
its <br>
> Copy On Write ("COW") design.=C2=A0 Writing a whole new file=
obviously <br>
> significantly increases the number of blocks which are written.=C2=A0 =
Requesting <br>
> that rsync only write to the file for the blocks which have changed re=
duces <br>
> the total number of blocks which get written.<br>
><br>
> The above helps quite a lot when using snapshots since then fewer bloc=
ks are <br>
> in the snapshots.<br>
><br>
> I have never tried --delete-during so I can't comment on that.<br>
><br>
> Bob<br>
> -- <br>
> Bob Friesenhahn<br>
> <a href=3D"mailto:bfriesen@simple.dallas.tx.us" target=3D"_blank" rel=
=3D"noreferrer">bfriesen@simple.dallas.tx.us</a>, <a href=3D"http://www.sim=
plesystems.org/users/bfriesen/" rel=3D"noreferrer noreferrer" target=3D"_bl=
ank">http://www.simplesystems.org/users/bfriesen/</a><br>;
> GraphicsMagick Maintainer,=C2=A0 =C2=A0 <a href=3D"http://www.Graphics=
Magick.org/" rel=3D"noreferrer noreferrer" target=3D"_blank">http://www.Gra=
phicsMagick.org/</a><br>
> Public Key,=C2=A0 =C2=A0 =C2=A0<a href=3D"http://www.simplesystems.org=
/users/bfriesen/public-key.txt" rel=3D"noreferrer noreferrer" target=3D"_bl=
ank">http://www.simplesystems.org/users/bfriesen/public-key.txt</a><br>;
><br>
><br>
<br>
<br>
----------------------------<br>
Andy Thomas,<br>
Time Domain Systems<br>
<br>
Tel: +44 (0)7866 556626<br>
<a href=3D"http://www.time-domain.co.uk" rel=3D"noreferrer noreferrer" targ=
et=3D"_blank">http://www.time-domain.co.uk</a><br>;
<br>
</blockquote></div>
--00000000000011651005eda74014--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOjFWZ6vxeXonEDndUvLkudRsRsBAd0sJ5ssOf4gLCgkVgSeyQ>