From owner-freebsd-fs@freebsd.org Tue Feb 28 19:45:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5938BCF1C58 for ; Tue, 28 Feb 2017 19:45:04 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2064BF2B; Tue, 28 Feb 2017 19:45:02 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1cinhj-0004z7-1B; Tue, 28 Feb 2017 20:44:59 +0100 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: =?utf-8?Q?Karli_Sj=C3=B6berg?= , "Gary Palmer" , "Shiva Bhanujan" Cc: "freebsd-fs@freebsd.org" , "Jeremy Faulkner" Subject: Re: FreeBSD restartable send/receive over WAN References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> Date: Tue, 28 Feb 2017 20:44:58 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: Quoted-Printable From: "Ronald Klop" Message-ID: In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> User-Agent: Opera Mail/12.16 (FreeBSD) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.2 X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED, BAYES_50 autolearn=disabled version=3.4.0 X-Scan-Signature: 326ae57122e7e55f093438bd1fe32346 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2017 19:45:04 -0000 On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan = wrote: > thanks for all the pointers for the compression algorithms. I ran a f= ew = > tests to compare compression overhead. These are local zfs = > send/receive, and no network is involved. > > time zfs send -v | | | zfs = > receive -s > > Here are the performance results that I got. > > no compression: > real 0m20.892s > user 0m0.000s > sys 0m5.587s > > xz -0: > real 8m38.569s > user 10m28.551s > sys 0m6.866s > > pxz -0: > real 4m38.448s > user 10m55.863s > sys 0m13.324s > > gzip: > real 3m51.297s > user 4m12.035s > sys 0m4.696s > > lz4: > real 0m29.912s > user 0m16.543s > sys 0m10.810s > > > lz4 has the least overhead in terms of time. pxz/xz seem to be = > prohibitive give the above results. Unless, there is something basic = = > I'm missing? > > I was really hoping that compressed sends would be available, as that = = > would actively eliminate this overhead, given that we use lz4 as the = > compression algorithm when writing to disks. Why don't you test this with a network in between? That would give much = = more valuable numbers to compare for your use case. The number above say nothing about the gain vs the bottleneck you are = trying to remove. Ronald. > > > ________________________________ > From: Karli Sj=C3=B6berg [karli.sjoberg@slu.se] > Sent: Sunday, February 26, 2017 8:41 AM > To: Gary Palmer > Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org > Subject: Re: FreeBSD restartable send/receive over WAN > > > Den 26 feb. 2017 4:16 em skrev Gary Palmer : >> >> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: >> > The compression that we use on our ZFS filesystems is lz4. So, if = I = >> have to pipe it through a compression algorithm, that'd be = >> uncompressing and compressing it 4 times. >> > >> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)= = >> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) >> > >> > isn't this quite expensive? We have to transfer multi terabyte fil= es = >> on a WAN link. I'm also of the understanding that gzip by itself is = = >> single-threaded, so that'd peg one of the CPUs to 100%. there might = be = >> other compression algorithms that can be used, but sending the ZFS as= = >> it is compressed on the filesystem is something that would be optimal= , = >> and would reduce the overhead of the additional [de]compressions that= = >> are taking place? >> >> Without going into the efficiency part of your message: >> >> archivers/pigz: Parallel GZIP >> archivers/pbzip2: Parallel BZIP2 >> archivers/pixz: Parallel, indexing version of XZ >> archivers/pxz: Parallel LZMA compressor using liblzma > > Also worth mentioning is, obviously: > archivers/lz4 > > :) > > /K > >> >> Regards, >> >> Gary >> >> > >> > ________________________________________ >> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] o= n = >> behalf of Jeremy Faulkner [gldisater@gmail.com] >> > Sent: Saturday, February 25, 2017 4:03 PM >> > To: freebsd-fs@freebsd.org >> > Subject: Re: FreeBSD restartable send/receive over WAN >> > >> > Pipe it through a compressor >> > >> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: >> > > Hi, >> > > >> > > I just tried restartable send/receive in 10.3 and it works like a= = >> charm. I was wondering if compressed send has made its way into = >> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the = >> -c/--compressed option. Any pointers? >> > > >> > > Regards, >> > > Shiva >> > > >> > > >> > > ________________________________________ >> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]= = >> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl] >> > > Sent: Thursday, February 16, 2017 10:41 AM >> > > To: freebsd-fs@freebsd.org >> > > Subject: Re: FreeBSD restartable send/receive over WAN >> > > >> > > On 2017-02-16 19:22, Shiva Bhanujan wrote: >> > >> Hello, >> > >> >> > >> I was wondering if restartable send/receive is available in = >> FreeBSD? We're running 10.2 and have a requirement of sending and = >> receiving ZFS snapshots over a WAN link. The snapshots could be more= = >> than a few terabytes. >> > >> >> > >> Can somebody please give me pointers, and if this feature is or = = >> isn't available in FreeBSD? >> > > >> > > FreeBSD 10.3 and later. >> > > >> > > _______________________________________________ >> > > freebsd-fs@freebsd.org mailing list >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > > To unsubscribe, send any mail to = >> "freebsd-fs-unsubscribe@freebsd.org" >> > > _______________________________________________ >> > > freebsd-fs@freebsd.org mailing list >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > > To unsubscribe, send any mail to = >> "freebsd-fs-unsubscribe@freebsd.org" >> > > >> > _______________________________________________ >> > freebsd-fs@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.or= g" >> > _______________________________________________ >> > freebsd-fs@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.or= g" >> > >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"