Date: Mon, 5 Jun 2017 21:06:41 +0000 From: Shiva Bhanujan <Shiva.Bhanujan@Quorum.net> To: Ronald Klop <ronald-lists@klop.ws>, =?iso-8859-1?Q?Karli_Sj=F6berg?= <karli.sjoberg@slu.se>, Gary Palmer <gpalmer@freebsd.org> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, Jeremy Faulkner <gldisater@gmail.com>, Shiva Bhanujan <Shiva.Bhanujan@Quorum.net> Subject: RE: FreeBSD restartable send/receive over WAN Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local> In-Reply-To: <op.ywmjkqczkndu52@53556c9c.cm-6-6b.dynamic.ziggo.nl> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <op.ywdyg82mkndu52@53556c9c.cm-6-6b.dynamic.ziggo.nl> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local>, <op.ywmjkqczkndu52@53556c9c.cm-6-6b.dynamic.ziggo.nl>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Ronald, Ideally, I'd like to send/receive the compressed blocks, and don't wish to compress/uncompress on the fly. Is compressed send/receive available on FreeBSD and if so, in what version? Regards, Shiva ________________________________________ From: Ronald Klop [ronald-lists@klop.ws] Sent: Sunday, March 05, 2017 3:01 AM To: Karli Sjöberg; Gary Palmer; Shiva Bhanujan Cc: freebsd-fs@freebsd.org; Jeremy Faulkner Subject: Re: FreeBSD restartable send/receive over WAN On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan <Shiva.Bhanujan@quorum.net> wrote: > I ran the same set of tests between 2 FreeBSD instances, connected on a > 1G LAN. The the comms was setup using mbuffer. Here's the basic > command. > > time zfs send -v <snapshot> | <compress> | mbuffer -O <IP>:8099 -b 1024 > -m 128M -P 10 -q -l /tmp/mbuffer.log > mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | <uncompress> > | zfs recv <snapshot> > > Here are the results that I got. > > no compression: > real 3m18.591s > user 0m0.390s > sys 0m8.177s > > xz -0: > real 7m26.349s > user 7m6.436s > sys 0m8.471s > > pxz -0: > real 2m28.470s > user 6m44.168s > sys 0m12.002s > > gzip: > real 3m12.482s > user 3m8.260s > sys 0m4.974s > > lz4: > real 1m58.363s > user 0m10.000s > sys 0m8.708s > > > > lz4 still comes out best. Is there anything else that I might be > missing in my tests? don't have a real setup at this time to test WAN > connections, but I'd like to think that these results can be > extrapolated to a WAN link also. Uhm, in your previous test no-compression came out best. I thought you wanted to test if sending on-disk compressed blocks over a network was a gain. Not a plain test of compression algoritms. As your current numbers show: the overhead of mbuffer.log is much higher than the overhead of compression+decompression for the lz4 case. real: 0m29 to 1m58. (All other compressors are actually faster with mbuffer in between? Probably because of more efficient buffering by mbuffer.) As the overhead of mbuffer is large it is a question how much gain you get from removing the extra compression step by sending the on-disk compressed blocks. I guess the time would go from 1m58 to about 1m30. NB: By splitting your zfs send/receive command in two, the numbers of time can be affected. Time prints the output as soon as zfs send ends. Have fun, Ronald. > ________________________________________ > From: Ronald Klop [ronald-lists@klop.ws] > Sent: Tuesday, February 28, 2017 11:44 AM > To: Karli Sjöberg; Gary Palmer; Shiva Bhanujan > Cc: freebsd-fs@freebsd.org; Jeremy Faulkner > Subject: Re: FreeBSD restartable send/receive over WAN > > On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan > <Shiva.Bhanujan@quorum.net> wrote: > >> thanks for all the pointers for the compression algorithms. I ran a few >> tests to compare compression overhead. These are local zfs >> send/receive, and no network is involved. >> >> time zfs send -v <src-snapshot> | <compress> | <uncompress> | zfs >> receive -s <dst-snapshot> >> >> Here are the performance results that I got. >> >> no compression: >> real 0m20.892s >> user 0m0.000s >> sys 0m5.587s >> >> xz -0: >> real 8m38.569s >> user 10m28.551s >> sys 0m6.866s >> >> pxz -0: >> real 4m38.448s >> user 10m55.863s >> sys 0m13.324s >> >> gzip: >> real 3m51.297s >> user 4m12.035s >> sys 0m4.696s >> >> lz4: >> real 0m29.912s >> user 0m16.543s >> sys 0m10.810s >> >> >> lz4 has the least overhead in terms of time. pxz/xz seem to be >> prohibitive give the above results. Unless, there is something basic >> I'm missing? >> >> I was really hoping that compressed sends would be available, as that >> would actively eliminate this overhead, given that we use lz4 as the >> compression algorithm when writing to disks. > > > Why don't you test this with a network in between? That would give much > more valuable numbers to compare for your use case. > The number above say nothing about the gain vs the bottleneck you are > trying to remove. > > Ronald. > > > > >> >> >> ________________________________ >> From: Karli Sjöberg [karli.sjoberg@slu.se] >> Sent: Sunday, February 26, 2017 8:41 AM >> To: Gary Palmer >> Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org >> Subject: Re: FreeBSD restartable send/receive over WAN >> >> >> Den 26 feb. 2017 4:16 em skrev Gary Palmer <gpalmer@freebsd.org>: >>> >>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: >>> > The compression that we use on our ZFS filesystems is lz4. So, if I >>> have to pipe it through a compression algorithm, that'd be >>> uncompressing and compressing it 4 times. >>> > >>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) >>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) >>> > >>> > isn't this quite expensive? We have to transfer multi terabyte files >>> on a WAN link. I'm also of the understanding that gzip by itself is >>> single-threaded, so that'd peg one of the CPUs to 100%. there might be >>> other compression algorithms that can be used, but sending the ZFS as >>> it is compressed on the filesystem is something that would be optimal, >>> and would reduce the overhead of the additional [de]compressions that >>> are taking place? >>> >>> Without going into the efficiency part of your message: >>> >>> archivers/pigz: Parallel GZIP >>> archivers/pbzip2: Parallel BZIP2 >>> archivers/pixz: Parallel, indexing version of XZ >>> archivers/pxz: Parallel LZMA compressor using liblzma >> >> Also worth mentioning is, obviously: >> archivers/lz4 >> >> :) >> >> /K >> >>> >>> Regards, >>> >>> Gary >>> >>> > >>> > ________________________________________ >>> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on >>> behalf of Jeremy Faulkner [gldisater@gmail.com] >>> > Sent: Saturday, February 25, 2017 4:03 PM >>> > To: freebsd-fs@freebsd.org >>> > Subject: Re: FreeBSD restartable send/receive over WAN >>> > >>> > Pipe it through a compressor >>> > >>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: >>> > > Hi, >>> > > >>> > > I just tried restartable send/receive in 10.3 and it works like a >>> charm. I was wondering if compressed send has made its way into >>> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the >>> -c/--compressed option. Any pointers? >>> > > >>> > > Regards, >>> > > Shiva >>> > > >>> > > >>> > > ________________________________________ >>> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] >>> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl] >>> > > Sent: Thursday, February 16, 2017 10:41 AM >>> > > To: freebsd-fs@freebsd.org >>> > > Subject: Re: FreeBSD restartable send/receive over WAN >>> > > >>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote: >>> > >> Hello, >>> > >> >>> > >> I was wondering if restartable send/receive is available in >>> FreeBSD? We're running 10.2 and have a requirement of sending and >>> receiving ZFS snapshots over a WAN link. The snapshots could be more >>> than a few terabytes. >>> > >> >>> > >> Can somebody please give me pointers, and if this feature is or >>> isn't available in FreeBSD? >>> > > >>> > > FreeBSD 10.3 and later. >>> > > >>> > > _______________________________________________ >>> > > freebsd-fs@freebsd.org mailing list >>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> > > To unsubscribe, send any mail to >>> "freebsd-fs-unsubscribe@freebsd.org" >>> > > _______________________________________________ >>> > > freebsd-fs@freebsd.org mailing list >>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> > > To unsubscribe, send any mail to >>> "freebsd-fs-unsubscribe@freebsd.org" >>> > > >>> > _______________________________________________ >>> > freebsd-fs@freebsd.org mailing list >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> > _______________________________________________ >>> > freebsd-fs@freebsd.org mailing list >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> > >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68>