From owner-freebsd-fs@freebsd.org Mon Jun 5 21:06:46 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C4F84B7A742 for ; Mon, 5 Jun 2017 21:06:46 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: from asp.reflexion.net (outbound-mail-210-5.reflexion.net [208.70.210.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 780856AE0F for ; Mon, 5 Jun 2017 21:06:45 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: (qmail 17401 invoked from network); 5 Jun 2017 21:06:44 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 5 Jun 2017 21:06:44 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Mon, 05 Jun 2017 17:06:44 -0400 (EDT) Received: (qmail 24447 invoked from network); 5 Jun 2017 21:06:43 -0000 Received: from unknown (HELO mail.quorumlabs.com) (64.74.133.216) by 0 (rfx-qmail) with (AES128-SHA encrypted) SMTP; 5 Jun 2017 21:06:43 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.02.0318.001; Mon, 5 Jun 2017 14:06:42 -0700 From: Shiva Bhanujan To: Ronald Klop , =?iso-8859-1?Q?Karli_Sj=F6berg?= , Gary Palmer CC: "freebsd-fs@freebsd.org" , Jeremy Faulkner , Shiva Bhanujan Subject: RE: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AQHSkE8fXkuYo0TvTyGgnDeYS4b6VKF+hfcfgADVzwCAAu69hYAEWqgAgJC1a0A= Date: Mon, 5 Jun 2017 21:06:41 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.74] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jun 2017 21:06:46 -0000 Hi Ronald,=0A= =0A= Ideally, I'd like to send/receive the compressed blocks, and don't wish to = compress/uncompress on the fly. Is compressed send/receive available on = FreeBSD and if so, in what version?=0A= =0A= Regards,=0A= Shiva=0A= =0A= =0A= ________________________________________=0A= From: Ronald Klop [ronald-lists@klop.ws]=0A= Sent: Sunday, March 05, 2017 3:01 AM=0A= To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= Subject: Re: FreeBSD restartable send/receive over WAN=0A= =0A= On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan=0A= wrote:=0A= =0A= > I ran the same set of tests between 2 FreeBSD instances, connected on a= =0A= > 1G LAN. The the comms was setup using mbuffer. Here's the basic=0A= > command.=0A= >=0A= > time zfs send -v | | mbuffer -O :8099 -b 1024= =0A= > -m 128M -P 10 -q -l /tmp/mbuffer.log=0A= > mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | = =0A= > | zfs recv =0A= >=0A= > Here are the results that I got.=0A= >=0A= > no compression:=0A= > real 3m18.591s=0A= > user 0m0.390s=0A= > sys 0m8.177s=0A= >=0A= > xz -0:=0A= > real 7m26.349s=0A= > user 7m6.436s=0A= > sys 0m8.471s=0A= >=0A= > pxz -0:=0A= > real 2m28.470s=0A= > user 6m44.168s=0A= > sys 0m12.002s=0A= >=0A= > gzip:=0A= > real 3m12.482s=0A= > user 3m8.260s=0A= > sys 0m4.974s=0A= >=0A= > lz4:=0A= > real 1m58.363s=0A= > user 0m10.000s=0A= > sys 0m8.708s=0A= >=0A= >=0A= >=0A= > lz4 still comes out best. Is there anything else that I might be=0A= > missing in my tests? don't have a real setup at this time to test WAN=0A= > connections, but I'd like to think that these results can be=0A= > extrapolated to a WAN link also.=0A= =0A= =0A= Uhm, in your previous test no-compression came out best. I thought you=0A= wanted to test if sending on-disk compressed blocks over a network was a=0A= gain. Not a plain test of compression algoritms.=0A= =0A= As your current numbers show: the overhead of mbuffer.log is much higher=0A= than the overhead of compression+decompression for the lz4 case. real:=0A= 0m29 to 1m58. (All other compressors are actually faster with mbuffer in=0A= between? Probably because of more efficient buffering by mbuffer.)=0A= As the overhead of mbuffer is large it is a question how much gain you get= =0A= from removing the extra compression step by sending the on-disk compressed= =0A= blocks.=0A= I guess the time would go from 1m58 to about 1m30.=0A= =0A= NB: By splitting your zfs send/receive command in two, the numbers of time= =0A= can be affected. Time prints the output as soon as zfs send ends.=0A= =0A= Have fun,=0A= Ronald.=0A= =0A= =0A= =0A= > ________________________________________=0A= > From: Ronald Klop [ronald-lists@klop.ws]=0A= > Sent: Tuesday, February 28, 2017 11:44 AM=0A= > To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= > Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >=0A= > On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan=0A= > wrote:=0A= >=0A= >> thanks for all the pointers for the compression algorithms. I ran a few= =0A= >> tests to compare compression overhead. These are local zfs=0A= >> send/receive, and no network is involved.=0A= >>=0A= >> time zfs send -v | | | zfs=0A= >> receive -s =0A= >>=0A= >> Here are the performance results that I got.=0A= >>=0A= >> no compression:=0A= >> real 0m20.892s=0A= >> user 0m0.000s=0A= >> sys 0m5.587s=0A= >>=0A= >> xz -0:=0A= >> real 8m38.569s=0A= >> user 10m28.551s=0A= >> sys 0m6.866s=0A= >>=0A= >> pxz -0:=0A= >> real 4m38.448s=0A= >> user 10m55.863s=0A= >> sys 0m13.324s=0A= >>=0A= >> gzip:=0A= >> real 3m51.297s=0A= >> user 4m12.035s=0A= >> sys 0m4.696s=0A= >>=0A= >> lz4:=0A= >> real 0m29.912s=0A= >> user 0m16.543s=0A= >> sys 0m10.810s=0A= >>=0A= >>=0A= >> lz4 has the least overhead in terms of time. pxz/xz seem to be=0A= >> prohibitive give the above results. Unless, there is something basic=0A= >> I'm missing?=0A= >>=0A= >> I was really hoping that compressed sends would be available, as that=0A= >> would actively eliminate this overhead, given that we use lz4 as the=0A= >> compression algorithm when writing to disks.=0A= >=0A= >=0A= > Why don't you test this with a network in between? That would give much= =0A= > more valuable numbers to compare for your use case.=0A= > The number above say nothing about the gain vs the bottleneck you are=0A= > trying to remove.=0A= >=0A= > Ronald.=0A= >=0A= >=0A= >=0A= >=0A= >>=0A= >>=0A= >> ________________________________=0A= >> From: Karli Sj=F6berg [karli.sjoberg@slu.se]=0A= >> Sent: Sunday, February 26, 2017 8:41 AM=0A= >> To: Gary Palmer=0A= >> Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org=0A= >> Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>=0A= >>=0A= >> Den 26 feb. 2017 4:16 em skrev Gary Palmer :=0A= >>>=0A= >>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:=0A= >>> > The compression that we use on our ZFS filesystems is lz4. So, if I= =0A= >>> have to pipe it through a compression algorithm, that'd be=0A= >>> uncompressing and compressing it 4 times.=0A= >>> >=0A= >>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)= =0A= >>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)=0A= >>> >=0A= >>> > isn't this quite expensive? We have to transfer multi terabyte files= =0A= >>> on a WAN link. I'm also of the understanding that gzip by itself is=0A= >>> single-threaded, so that'd peg one of the CPUs to 100%. there might be= =0A= >>> other compression algorithms that can be used, but sending the ZFS as= =0A= >>> it is compressed on the filesystem is something that would be optimal,= =0A= >>> and would reduce the overhead of the additional [de]compressions that= =0A= >>> are taking place?=0A= >>>=0A= >>> Without going into the efficiency part of your message:=0A= >>>=0A= >>> archivers/pigz: Parallel GZIP=0A= >>> archivers/pbzip2: Parallel BZIP2=0A= >>> archivers/pixz: Parallel, indexing version of XZ=0A= >>> archivers/pxz: Parallel LZMA compressor using liblzma=0A= >>=0A= >> Also worth mentioning is, obviously:=0A= >> archivers/lz4=0A= >>=0A= >> :)=0A= >>=0A= >> /K=0A= >>=0A= >>>=0A= >>> Regards,=0A= >>>=0A= >>> Gary=0A= >>>=0A= >>> >=0A= >>> > ________________________________________=0A= >>> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on= =0A= >>> behalf of Jeremy Faulkner [gldisater@gmail.com]=0A= >>> > Sent: Saturday, February 25, 2017 4:03 PM=0A= >>> > To: freebsd-fs@freebsd.org=0A= >>> > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>> >=0A= >>> > Pipe it through a compressor=0A= >>> >=0A= >>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:=0A= >>> > > Hi,=0A= >>> > >=0A= >>> > > I just tried restartable send/receive in 10.3 and it works like a= =0A= >>> charm. I was wondering if compressed send has made its way into=0A= >>> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the=0A= >>> -c/--compressed option. Any pointers?=0A= >>> > >=0A= >>> > > Regards,=0A= >>> > > Shiva=0A= >>> > >=0A= >>> > >=0A= >>> > > ________________________________________=0A= >>> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]= =0A= >>> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl]=0A= >>> > > Sent: Thursday, February 16, 2017 10:41 AM=0A= >>> > > To: freebsd-fs@freebsd.org=0A= >>> > > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>> > >=0A= >>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:=0A= >>> > >> Hello,=0A= >>> > >>=0A= >>> > >> I was wondering if restartable send/receive is available in=0A= >>> FreeBSD? We're running 10.2 and have a requirement of sending and=0A= >>> receiving ZFS snapshots over a WAN link. The snapshots could be more= =0A= >>> than a few terabytes.=0A= >>> > >>=0A= >>> > >> Can somebody please give me pointers, and if this feature is or=0A= >>> isn't available in FreeBSD?=0A= >>> > >=0A= >>> > > FreeBSD 10.3 and later.=0A= >>> > >=0A= >>> > > _______________________________________________=0A= >>> > > freebsd-fs@freebsd.org mailing list=0A= >>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > > To unsubscribe, send any mail to=0A= >>> "freebsd-fs-unsubscribe@freebsd.org"=0A= >>> > > _______________________________________________=0A= >>> > > freebsd-fs@freebsd.org mailing list=0A= >>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > > To unsubscribe, send any mail to=0A= >>> "freebsd-fs-unsubscribe@freebsd.org"=0A= >>> > >=0A= >>> > _______________________________________________=0A= >>> > freebsd-fs@freebsd.org mailing list=0A= >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >>> > _______________________________________________=0A= >>> > freebsd-fs@freebsd.org mailing list=0A= >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >>> >=0A= >>> _______________________________________________=0A= >>> freebsd-fs@freebsd.org mailing list=0A= >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> _______________________________________________=0A= >> freebsd-fs@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A=