Date: Fri, 3 Mar 2017 00:34:57 +0000 From: Shiva Bhanujan <Shiva.Bhanujan@Quorum.net> To: Ronald Klop <ronald-lists@klop.ws>, =?iso-8859-1?Q?Karli_Sj=F6berg?= <karli.sjoberg@slu.se>, Gary Palmer <gpalmer@freebsd.org> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, Jeremy Faulkner <gldisater@gmail.com> Subject: RE: FreeBSD restartable send/receive over WAN Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> In-Reply-To: <op.ywdyg82mkndu52@53556c9c.cm-6-6b.dynamic.ziggo.nl> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local>, <op.ywdyg82mkndu52@53556c9c.cm-6-6b.dynamic.ziggo.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
I ran the same set of tests between 2 FreeBSD instances, connected on a 1G = LAN. The the comms was setup using mbuffer. Here's the basic command.=0A= =0A= time zfs send -v <snapshot> | <compress> | mbuffer -O <IP>:8099 -b 1024 -m = 128M -P 10 -q -l /tmp/mbuffer.log=0A= mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | <uncompress> | = zfs recv <snapshot>=0A= =0A= Here are the results that I got.=0A= =0A= no compression:=0A= real 3m18.591s=0A= user 0m0.390s=0A= sys 0m8.177s=0A= =0A= xz -0:=0A= real 7m26.349s=0A= user 7m6.436s=0A= sys 0m8.471s=0A= =0A= pxz -0:=0A= real 2m28.470s=0A= user 6m44.168s=0A= sys 0m12.002s=0A= =0A= gzip:=0A= real 3m12.482s=0A= user 3m8.260s=0A= sys 0m4.974s=0A= =0A= lz4:=0A= real 1m58.363s=0A= user 0m10.000s=0A= sys 0m8.708s=0A= =0A= =0A= =0A= lz4 still comes out best. Is there anything else that I might be missing i= n my tests? don't have a real setup at this time to test WAN connections, = but I'd like to think that these results can be extrapolated to a WAN link = also. =0A= =0A= =0A= ________________________________________=0A= From: Ronald Klop [ronald-lists@klop.ws]=0A= Sent: Tuesday, February 28, 2017 11:44 AM=0A= To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= Subject: Re: FreeBSD restartable send/receive over WAN=0A= =0A= On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan=0A= <Shiva.Bhanujan@quorum.net> wrote:=0A= =0A= > thanks for all the pointers for the compression algorithms. I ran a few= =0A= > tests to compare compression overhead. These are local zfs=0A= > send/receive, and no network is involved.=0A= >=0A= > time zfs send -v <src-snapshot> | <compress> | <uncompress> | zfs=0A= > receive -s <dst-snapshot>=0A= >=0A= > Here are the performance results that I got.=0A= >=0A= > no compression:=0A= > real 0m20.892s=0A= > user 0m0.000s=0A= > sys 0m5.587s=0A= >=0A= > xz -0:=0A= > real 8m38.569s=0A= > user 10m28.551s=0A= > sys 0m6.866s=0A= >=0A= > pxz -0:=0A= > real 4m38.448s=0A= > user 10m55.863s=0A= > sys 0m13.324s=0A= >=0A= > gzip:=0A= > real 3m51.297s=0A= > user 4m12.035s=0A= > sys 0m4.696s=0A= >=0A= > lz4:=0A= > real 0m29.912s=0A= > user 0m16.543s=0A= > sys 0m10.810s=0A= >=0A= >=0A= > lz4 has the least overhead in terms of time. pxz/xz seem to be=0A= > prohibitive give the above results. Unless, there is something basic=0A= > I'm missing?=0A= >=0A= > I was really hoping that compressed sends would be available, as that=0A= > would actively eliminate this overhead, given that we use lz4 as the=0A= > compression algorithm when writing to disks.=0A= =0A= =0A= Why don't you test this with a network in between? That would give much=0A= more valuable numbers to compare for your use case.=0A= The number above say nothing about the gain vs the bottleneck you are=0A= trying to remove.=0A= =0A= Ronald.=0A= =0A= =0A= =0A= =0A= >=0A= >=0A= > ________________________________=0A= > From: Karli Sj=F6berg [karli.sjoberg@slu.se]=0A= > Sent: Sunday, February 26, 2017 8:41 AM=0A= > To: Gary Palmer=0A= > Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org=0A= > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >=0A= >=0A= > Den 26 feb. 2017 4:16 em skrev Gary Palmer <gpalmer@freebsd.org>:=0A= >>=0A= >> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:=0A= >> > The compression that we use on our ZFS filesystems is lz4. So, if I= =0A= >> have to pipe it through a compression algorithm, that'd be=0A= >> uncompressing and compressing it 4 times.=0A= >> >=0A= >> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)=0A= >> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)=0A= >> >=0A= >> > isn't this quite expensive? We have to transfer multi terabyte files= =0A= >> on a WAN link. I'm also of the understanding that gzip by itself is=0A= >> single-threaded, so that'd peg one of the CPUs to 100%. there might be= =0A= >> other compression algorithms that can be used, but sending the ZFS as=0A= >> it is compressed on the filesystem is something that would be optimal,= =0A= >> and would reduce the overhead of the additional [de]compressions that=0A= >> are taking place?=0A= >>=0A= >> Without going into the efficiency part of your message:=0A= >>=0A= >> archivers/pigz: Parallel GZIP=0A= >> archivers/pbzip2: Parallel BZIP2=0A= >> archivers/pixz: Parallel, indexing version of XZ=0A= >> archivers/pxz: Parallel LZMA compressor using liblzma=0A= >=0A= > Also worth mentioning is, obviously:=0A= > archivers/lz4=0A= >=0A= > :)=0A= >=0A= > /K=0A= >=0A= >>=0A= >> Regards,=0A= >>=0A= >> Gary=0A= >>=0A= >> >=0A= >> > ________________________________________=0A= >> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on= =0A= >> behalf of Jeremy Faulkner [gldisater@gmail.com]=0A= >> > Sent: Saturday, February 25, 2017 4:03 PM=0A= >> > To: freebsd-fs@freebsd.org=0A= >> > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >> >=0A= >> > Pipe it through a compressor=0A= >> >=0A= >> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:=0A= >> > > Hi,=0A= >> > >=0A= >> > > I just tried restartable send/receive in 10.3 and it works like a=0A= >> charm. I was wondering if compressed send has made its way into=0A= >> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the=0A= >> -c/--compressed option. Any pointers?=0A= >> > >=0A= >> > > Regards,=0A= >> > > Shiva=0A= >> > >=0A= >> > >=0A= >> > > ________________________________________=0A= >> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]=0A= >> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl]=0A= >> > > Sent: Thursday, February 16, 2017 10:41 AM=0A= >> > > To: freebsd-fs@freebsd.org=0A= >> > > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >> > >=0A= >> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:=0A= >> > >> Hello,=0A= >> > >>=0A= >> > >> I was wondering if restartable send/receive is available in=0A= >> FreeBSD? We're running 10.2 and have a requirement of sending and=0A= >> receiving ZFS snapshots over a WAN link. The snapshots could be more=0A= >> than a few terabytes.=0A= >> > >>=0A= >> > >> Can somebody please give me pointers, and if this feature is or=0A= >> isn't available in FreeBSD?=0A= >> > >=0A= >> > > FreeBSD 10.3 and later.=0A= >> > >=0A= >> > > _______________________________________________=0A= >> > > freebsd-fs@freebsd.org mailing list=0A= >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > > To unsubscribe, send any mail to=0A= >> "freebsd-fs-unsubscribe@freebsd.org"=0A= >> > > _______________________________________________=0A= >> > > freebsd-fs@freebsd.org mailing list=0A= >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > > To unsubscribe, send any mail to=0A= >> "freebsd-fs-unsubscribe@freebsd.org"=0A= >> > >=0A= >> > _______________________________________________=0A= >> > freebsd-fs@freebsd.org mailing list=0A= >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> > _______________________________________________=0A= >> > freebsd-fs@freebsd.org mailing list=0A= >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> >=0A= >> _______________________________________________=0A= >> freebsd-fs@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= > _______________________________________________=0A= > freebsd-fs@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA>