Date: Mon, 5 Jun 2017 21:06:41 +0000 From: Shiva Bhanujan <Shiva.Bhanujan@Quorum.net> To: Ronald Klop <ronald-lists@klop.ws>, =?iso-8859-1?Q?Karli_Sj=F6berg?= <karli.sjoberg@slu.se>, Gary Palmer <gpalmer@freebsd.org> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, Jeremy Faulkner <gldisater@gmail.com>, Shiva Bhanujan <Shiva.Bhanujan@Quorum.net> Subject: RE: FreeBSD restartable send/receive over WAN Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local> In-Reply-To: <op.ywmjkqczkndu52@53556c9c.cm-6-6b.dynamic.ziggo.nl> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <op.ywdyg82mkndu52@53556c9c.cm-6-6b.dynamic.ziggo.nl> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local>, <op.ywmjkqczkndu52@53556c9c.cm-6-6b.dynamic.ziggo.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Ronald,=0A= =0A= Ideally, I'd like to send/receive the compressed blocks, and don't wish to = compress/uncompress on the fly. Is compressed send/receive available on = FreeBSD and if so, in what version?=0A= =0A= Regards,=0A= Shiva=0A= =0A= =0A= ________________________________________=0A= From: Ronald Klop [ronald-lists@klop.ws]=0A= Sent: Sunday, March 05, 2017 3:01 AM=0A= To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= Subject: Re: FreeBSD restartable send/receive over WAN=0A= =0A= On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan=0A= <Shiva.Bhanujan@quorum.net> wrote:=0A= =0A= > I ran the same set of tests between 2 FreeBSD instances, connected on a= =0A= > 1G LAN. The the comms was setup using mbuffer. Here's the basic=0A= > command.=0A= >=0A= > time zfs send -v <snapshot> | <compress> | mbuffer -O <IP>:8099 -b 1024= =0A= > -m 128M -P 10 -q -l /tmp/mbuffer.log=0A= > mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | <uncompress>= =0A= > | zfs recv <snapshot>=0A= >=0A= > Here are the results that I got.=0A= >=0A= > no compression:=0A= > real 3m18.591s=0A= > user 0m0.390s=0A= > sys 0m8.177s=0A= >=0A= > xz -0:=0A= > real 7m26.349s=0A= > user 7m6.436s=0A= > sys 0m8.471s=0A= >=0A= > pxz -0:=0A= > real 2m28.470s=0A= > user 6m44.168s=0A= > sys 0m12.002s=0A= >=0A= > gzip:=0A= > real 3m12.482s=0A= > user 3m8.260s=0A= > sys 0m4.974s=0A= >=0A= > lz4:=0A= > real 1m58.363s=0A= > user 0m10.000s=0A= > sys 0m8.708s=0A= >=0A= >=0A= >=0A= > lz4 still comes out best. Is there anything else that I might be=0A= > missing in my tests? don't have a real setup at this time to test WAN=0A= > connections, but I'd like to think that these results can be=0A= > extrapolated to a WAN link also.=0A= =0A= =0A= Uhm, in your previous test no-compression came out best. I thought you=0A= wanted to test if sending on-disk compressed blocks over a network was a=0A= gain. Not a plain test of compression algoritms.=0A= =0A= As your current numbers show: the overhead of mbuffer.log is much higher=0A= than the overhead of compression+decompression for the lz4 case. real:=0A= 0m29 to 1m58. (All other compressors are actually faster with mbuffer in=0A= between? Probably because of more efficient buffering by mbuffer.)=0A= As the overhead of mbuffer is large it is a question how much gain you get= =0A= from removing the extra compression step by sending the on-disk compressed= =0A= blocks.=0A= I guess the time would go from 1m58 to about 1m30.=0A= =0A= NB: By splitting your zfs send/receive command in two, the numbers of time= =0A= can be affected. Time prints the output as soon as zfs send ends.=0A= =0A= Have fun,=0A= Ronald.=0A= =0A= =0A= =0A= > ________________________________________=0A= > From: Ronald Klop [ronald-lists@klop.ws]=0A= > Sent: Tuesday, February 28, 2017 11:44 AM=0A= > To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= > Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >=0A= > On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan=0A= > <Shiva.Bhanujan@quorum.net> wrote:=0A= >=0A= >> thanks for all the pointers for the compression algorithms. I ran a few= =0A= >> tests to compare compression overhead. These are local zfs=0A= >> send/receive, and no network is involved.=0A= >>=0A= >> time zfs send -v <src-snapshot> | <compress> | <uncompress> | zfs=0A= >> receive -s <dst-snapshot>=0A= >>=0A= >> Here are the performance results that I got.=0A= >>=0A= >> no compression:=0A= >> real 0m20.892s=0A= >> user 0m0.000s=0A= >> sys 0m5.587s=0A= >>=0A= >> xz -0:=0A= >> real 8m38.569s=0A= >> user 10m28.551s=0A= >> sys 0m6.866s=0A= >>=0A= >> pxz -0:=0A= >> real 4m38.448s=0A= >> user 10m55.863s=0A= >> sys 0m13.324s=0A= >>=0A= >> gzip:=0A= >> real 3m51.297s=0A= >> user 4m12.035s=0A= >> sys 0m4.696s=0A= >>=0A= >> lz4:=0A= >> real 0m29.912s=0A= >> user 0m16.543s=0A= >> sys 0m10.810s=0A= >>=0A= >>=0A= >> lz4 has the least overhead in terms of time. pxz/xz seem to be=0A= >> prohibitive give the above results. Unless, there is something basic=0A= >> I'm missing?=0A= >>=0A= >> I was really hoping that compressed sends would be available, as that=0A= >> would actively eliminate this overhead, given that we use lz4 as the=0A= >> compression algorithm when writing to disks.=0A= >=0A= >=0A= > Why don't you test this with a network in between? That would give much= =0A= > more valuable numbers to compare for your use case.=0A= > The number above say nothing about the gain vs the bottleneck you are=0A= > trying to remove.=0A= >=0A= > Ronald.=0A= >=0A= >=0A= >=0A= >=0A= >>=0A= >>=0A= >> ________________________________=0A= >> From: Karli Sj=F6berg [karli.sjoberg@slu.se]=0A= >> Sent: Sunday, February 26, 2017 8:41 AM=0A= >> To: Gary Palmer=0A= >> Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org=0A= >> Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>=0A= >>=0A= >> Den 26 feb. 2017 4:16 em skrev Gary Palmer <gpalmer@freebsd.org>:=0A= >>>=0A= >>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:=0A= >>> > The compression that we use on our ZFS filesystems is lz4. So, if I= =0A= >>> have to pipe it through a compression algorithm, that'd be=0A= >>> uncompressing and compressing it 4 times.=0A= >>> >=0A= >>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)= =0A= >>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)=0A= >>> >=0A= >>> > isn't this quite expensive? We have to transfer multi terabyte files= =0A= >>> on a WAN link. I'm also of the understanding that gzip by itself is=0A= >>> single-threaded, so that'd peg one of the CPUs to 100%. there might be= =0A= >>> other compression algorithms that can be used, but sending the ZFS as= =0A= >>> it is compressed on the filesystem is something that would be optimal,= =0A= >>> and would reduce the overhead of the additional [de]compressions that= =0A= >>> are taking place?=0A= >>>=0A= >>> Without going into the efficiency part of your message:=0A= >>>=0A= >>> archivers/pigz: Parallel GZIP=0A= >>> archivers/pbzip2: Parallel BZIP2=0A= >>> archivers/pixz: Parallel, indexing version of XZ=0A= >>> archivers/pxz: Parallel LZMA compressor using liblzma=0A= >>=0A= >> Also worth mentioning is, obviously:=0A= >> archivers/lz4=0A= >>=0A= >> :)=0A= >>=0A= >> /K=0A= >>=0A= >>>=0A= >>> Regards,=0A= >>>=0A= >>> Gary=0A= >>>=0A= >>> >=0A= >>> > ________________________________________=0A= >>> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on= =0A= >>> behalf of Jeremy Faulkner [gldisater@gmail.com]=0A= >>> > Sent: Saturday, February 25, 2017 4:03 PM=0A= >>> > To: freebsd-fs@freebsd.org=0A= >>> > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>> >=0A= >>> > Pipe it through a compressor=0A= >>> >=0A= >>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:=0A= >>> > > Hi,=0A= >>> > >=0A= >>> > > I just tried restartable send/receive in 10.3 and it works like a= =0A= >>> charm. I was wondering if compressed send has made its way into=0A= >>> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the=0A= >>> -c/--compressed option. Any pointers?=0A= >>> > >=0A= >>> > > Regards,=0A= >>> > > Shiva=0A= >>> > >=0A= >>> > >=0A= >>> > > ________________________________________=0A= >>> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]= =0A= >>> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl]=0A= >>> > > Sent: Thursday, February 16, 2017 10:41 AM=0A= >>> > > To: freebsd-fs@freebsd.org=0A= >>> > > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>> > >=0A= >>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:=0A= >>> > >> Hello,=0A= >>> > >>=0A= >>> > >> I was wondering if restartable send/receive is available in=0A= >>> FreeBSD? We're running 10.2 and have a requirement of sending and=0A= >>> receiving ZFS snapshots over a WAN link. The snapshots could be more= =0A= >>> than a few terabytes.=0A= >>> > >>=0A= >>> > >> Can somebody please give me pointers, and if this feature is or=0A= >>> isn't available in FreeBSD?=0A= >>> > >=0A= >>> > > FreeBSD 10.3 and later.=0A= >>> > >=0A= >>> > > _______________________________________________=0A= >>> > > freebsd-fs@freebsd.org mailing list=0A= >>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > > To unsubscribe, send any mail to=0A= >>> "freebsd-fs-unsubscribe@freebsd.org"=0A= >>> > > _______________________________________________=0A= >>> > > freebsd-fs@freebsd.org mailing list=0A= >>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > > To unsubscribe, send any mail to=0A= >>> "freebsd-fs-unsubscribe@freebsd.org"=0A= >>> > >=0A= >>> > _______________________________________________=0A= >>> > freebsd-fs@freebsd.org mailing list=0A= >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >>> > _______________________________________________=0A= >>> > freebsd-fs@freebsd.org mailing list=0A= >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >>> >=0A= >>> _______________________________________________=0A= >>> freebsd-fs@freebsd.org mailing list=0A= >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> _______________________________________________=0A= >> freebsd-fs@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68>