From owner-freebsd-fs@freebsd.org Fri Mar 3 00:35:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A30CDCF6F5E for ; Fri, 3 Mar 2017 00:35:04 +0000 (UTC) (envelope-from Shiva.Bhanujan@Quorum.net) Received: from mail.quorumlabs.com (mail.quorum.net [64.74.133.216]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "mail.quorumlabs.com", Issuer "mail.quorumlabs.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 90B93198B; Fri, 3 Mar 2017 00:35:03 +0000 (UTC) (envelope-from Shiva.Bhanujan@Quorum.net) Received: from QLEXC01.Quorum.local ([fe80::79eb:fde8:ed8e:5648]) by QLEXC01.Quorum.local ([fe80::79eb:fde8:ed8e:5648%14]) with mapi id 14.02.0318.001; Thu, 2 Mar 2017 16:34:57 -0800 From: Shiva Bhanujan To: Ronald Klop , =?iso-8859-1?Q?Karli_Sj=F6berg?= , Gary Palmer CC: "freebsd-fs@freebsd.org" , Jeremy Faulkner Subject: RE: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AQHSkE8fXkuYo0TvTyGgnDeYS4b6VKF+hfcfgADVzwCAAu69hQ== Date: Fri, 3 Mar 2017 00:34:57 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.75] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 00:35:04 -0000 I ran the same set of tests between 2 FreeBSD instances, connected on a 1G = LAN. The the comms was setup using mbuffer. Here's the basic command.=0A= =0A= time zfs send -v | | mbuffer -O :8099 -b 1024 -m = 128M -P 10 -q -l /tmp/mbuffer.log=0A= mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | | = zfs recv =0A= =0A= Here are the results that I got.=0A= =0A= no compression:=0A= real 3m18.591s=0A= user 0m0.390s=0A= sys 0m8.177s=0A= =0A= xz -0:=0A= real 7m26.349s=0A= user 7m6.436s=0A= sys 0m8.471s=0A= =0A= pxz -0:=0A= real 2m28.470s=0A= user 6m44.168s=0A= sys 0m12.002s=0A= =0A= gzip:=0A= real 3m12.482s=0A= user 3m8.260s=0A= sys 0m4.974s=0A= =0A= lz4:=0A= real 1m58.363s=0A= user 0m10.000s=0A= sys 0m8.708s=0A= =0A= =0A= =0A= lz4 still comes out best. Is there anything else that I might be missing i= n my tests? don't have a real setup at this time to test WAN connections, = but I'd like to think that these results can be extrapolated to a WAN link = also. =0A= =0A= =0A= ________________________________________=0A= From: Ronald Klop [ronald-lists@klop.ws]=0A= Sent: Tuesday, February 28, 2017 11:44 AM=0A= To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= Subject: Re: FreeBSD restartable send/receive over WAN=0A= =0A= On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan=0A= wrote:=0A= =0A= > thanks for all the pointers for the compression algorithms. I ran a few= =0A= > tests to compare compression overhead. These are local zfs=0A= > send/receive, and no network is involved.=0A= >=0A= > time zfs send -v | | | zfs=0A= > receive -s =0A= >=0A= > Here are the performance results that I got.=0A= >=0A= > no compression:=0A= > real 0m20.892s=0A= > user 0m0.000s=0A= > sys 0m5.587s=0A= >=0A= > xz -0:=0A= > real 8m38.569s=0A= > user 10m28.551s=0A= > sys 0m6.866s=0A= >=0A= > pxz -0:=0A= > real 4m38.448s=0A= > user 10m55.863s=0A= > sys 0m13.324s=0A= >=0A= > gzip:=0A= > real 3m51.297s=0A= > user 4m12.035s=0A= > sys 0m4.696s=0A= >=0A= > lz4:=0A= > real 0m29.912s=0A= > user 0m16.543s=0A= > sys 0m10.810s=0A= >=0A= >=0A= > lz4 has the least overhead in terms of time. pxz/xz seem to be=0A= > prohibitive give the above results. Unless, there is something basic=0A= > I'm missing?=0A= >=0A= > I was really hoping that compressed sends would be available, as that=0A= > would actively eliminate this overhead, given that we use lz4 as the=0A= > compression algorithm when writing to disks.=0A= =0A= =0A= Why don't you test this with a network in between? That would give much=0A= more valuable numbers to compare for your use case.=0A= The number above say nothing about the gain vs the bottleneck you are=0A= trying to remove.=0A= =0A= Ronald.=0A= =0A= =0A= =0A= =0A= >=0A= >=0A= > ________________________________=0A= > From: Karli Sj=F6berg [karli.sjoberg@slu.se]=0A= > Sent: Sunday, February 26, 2017 8:41 AM=0A= > To: Gary Palmer=0A= > Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org=0A= > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >=0A= >=0A= > Den 26 feb. 2017 4:16 em skrev Gary Palmer :=0A= >>=0A= >> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:=0A= >> > The compression that we use on our ZFS filesystems is lz4. So, if I= =0A= >> have to pipe it through a compression algorithm, that'd be=0A= >> uncompressing and compressing it 4 times.=0A= >> >=0A= >> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)=0A= >> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)=0A= >> >=0A= >> > isn't this quite expensive? We have to transfer multi terabyte files= =0A= >> on a WAN link. I'm also of the understanding that gzip by itself is=0A= >> single-threaded, so that'd peg one of the CPUs to 100%. there might be= =0A= >> other compression algorithms that can be used, but sending the ZFS as=0A= >> it is compressed on the filesystem is something that would be optimal,= =0A= >> and would reduce the overhead of the additional [de]compressions that=0A= >> are taking place?=0A= >>=0A= >> Without going into the efficiency part of your message:=0A= >>=0A= >> archivers/pigz: Parallel GZIP=0A= >> archivers/pbzip2: Parallel BZIP2=0A= >> archivers/pixz: Parallel, indexing version of XZ=0A= >> archivers/pxz: Parallel LZMA compressor using liblzma=0A= >=0A= > Also worth mentioning is, obviously:=0A= > archivers/lz4=0A= >=0A= > :)=0A= >=0A= > /K=0A= >=0A= >>=0A= >> Regards,=0A= >>=0A= >> Gary=0A= >>=0A= >> >=0A= >> > ________________________________________=0A= >> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on= =0A= >> behalf of Jeremy Faulkner [gldisater@gmail.com]=0A= >> > Sent: Saturday, February 25, 2017 4:03 PM=0A= >> > To: freebsd-fs@freebsd.org=0A= >> > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >> >=0A= >> > Pipe it through a compressor=0A= >> >=0A= >> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:=0A= >> > > Hi,=0A= >> > >=0A= >> > > I just tried restartable send/receive in 10.3 and it works like a=0A= >> charm. I was wondering if compressed send has made its way into=0A= >> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the=0A= >> -c/--compressed option. Any pointers?=0A= >> > >=0A= >> > > Regards,=0A= >> > > Shiva=0A= >> > >=0A= >> > >=0A= >> > > ________________________________________=0A= >> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]=0A= >> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl]=0A= >> > > Sent: Thursday, February 16, 2017 10:41 AM=0A= >> > > To: freebsd-fs@freebsd.org=0A= >> > > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >> > >=0A= >> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:=0A= >> > >> Hello,=0A= >> > >>=0A= >> > >> I was wondering if restartable send/receive is available in=0A= >> FreeBSD? We're running 10.2 and have a requirement of sending and=0A= >> receiving ZFS snapshots over a WAN link. The snapshots could be more=0A= >> than a few terabytes.=0A= >> > >>=0A= >> > >> Can somebody please give me pointers, and if this feature is or=0A= >> isn't available in FreeBSD?=0A= >> > >=0A= >> > > FreeBSD 10.3 and later.=0A= >> > >=0A= >> > > _______________________________________________=0A= >> > > freebsd-fs@freebsd.org mailing list=0A= >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > > To unsubscribe, send any mail to=0A= >> "freebsd-fs-unsubscribe@freebsd.org"=0A= >> > > _______________________________________________=0A= >> > > freebsd-fs@freebsd.org mailing list=0A= >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > > To unsubscribe, send any mail to=0A= >> "freebsd-fs-unsubscribe@freebsd.org"=0A= >> > >=0A= >> > _______________________________________________=0A= >> > freebsd-fs@freebsd.org mailing list=0A= >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> > _______________________________________________=0A= >> > freebsd-fs@freebsd.org mailing list=0A= >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> >=0A= >> _______________________________________________=0A= >> freebsd-fs@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= > _______________________________________________=0A= > freebsd-fs@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A=