From owner-freebsd-fs@freebsd.org Tue Jun 13 23:34:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A293AC09546 for ; Tue, 13 Jun 2017 23:34:17 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: from asp.reflexion.net (outbound-mail-210-12.reflexion.net [208.70.210.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5F3F370B32 for ; Tue, 13 Jun 2017 23:34:16 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: (qmail 23155 invoked from network); 13 Jun 2017 23:34:15 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 13 Jun 2017 23:34:15 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Tue, 13 Jun 2017 19:34:15 -0400 (EDT) Received: (qmail 11043 invoked from network); 13 Jun 2017 23:34:15 -0000 Received: from unknown (HELO mail.quorumlabs.com) (64.74.133.216) by 0 (rfx-qmail) with (AES128-SHA encrypted) SMTP; 13 Jun 2017 23:34:15 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.02.0318.001; Tue, 13 Jun 2017 16:34:13 -0700 From: Shiva Bhanujan To: Ronald Klop , =?iso-8859-1?Q?Karli_Sj=F6berg?= , Gary Palmer CC: "freebsd-fs@freebsd.org" , Jeremy Faulkner , Shiva Bhanujan Subject: RE: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AQHSkE8fXkuYo0TvTyGgnDeYS4b6VKF+hfcfgADVzwCAAu69hYAEWqgAgJC1a0CAAIDAAIAMPuzC Date: Tue, 13 Jun 2017 23:34:12 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35DA475@QLEXC01.Quorum.local> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.65] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2017 23:34:17 -0000 Hi Ronald, thanks for the info. I'm going to work w/ FreeBSD 11.0, as it has = restartable send/receive. the compressed send is desirable though. Regards, Shiva ________________________________ From: Ronald Klop =5Bronald-lists=40klop.ws=5D Sent: Monday, June 05, 2017 2:33 PM To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan Cc: freebsd-fs=40freebsd.org; Jeremy Faulkner Subject: Re: FreeBSD restartable send/receive over WAN Hi, I'm not authoritative about this, but I see the feature in FreeBSD 12: https://www.freebsd.org/cgi/man.cgi?query=3Dzfs&apropos=3D0&sektion=3D8&man= path=3DFreeBSD+12-current&arch=3Ddefault&format=3Dhtml (option -c to 'zfs send') But not in FreeBSD 11. https://www.freebsd.org/cgi/man.cgi?query=3Dzfs&apropos=3D0&sektion=3D8&man= path=3DFreeBSD+11-current&arch=3Ddefault&format=3Dhtml Regards, Ronald. On Mon, 05 Jun 2017 23:06:41 +0200, Shiva Bhanujan > wrote: > Hi Ronald, > > Ideally, I'd like to send/receive the compressed blocks, and don't wish > to compress/uncompress on the fly. Is compressed send/receive > available on FreeBSD and if so, in what version? > > Regards, > Shiva > > > ________________________________________ > From: Ronald Klop = =5Bronald-lists=40klop.ws=5D > Sent: Sunday, March 05, 2017 3:01 AM > To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan > Cc: freebsd-fs=40freebsd.org; Jeremy = Faulkner > Subject: Re: FreeBSD restartable send/receive over WAN > > On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan > > wrote: > >> I ran the same set of tests between 2 FreeBSD instances, connected on a >> 1G LAN. The the comms was setup using mbuffer. Here's the basic >> command. >> >> time zfs send -v =7C =7C mbuffer -O :8099 -b = 1024 >> -m 128M -P 10 -q -l /tmp/mbuffer.log >> mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log =7C = >> =7C zfs recv >> >> Here are the results that I got. >> >> no compression: >> real 3m18.591s >> user 0m0.390s >> sys 0m8.177s >> >> xz -0: >> real 7m26.349s >> user 7m6.436s >> sys 0m8.471s >> >> pxz -0: >> real 2m28.470s >> user 6m44.168s >> sys 0m12.002s >> >> gzip: >> real 3m12.482s >> user 3m8.260s >> sys 0m4.974s >> >> lz4: >> real 1m58.363s >> user 0m10.000s >> sys 0m8.708s >> >> >> >> lz4 still comes out best. Is there anything else that I might be >> missing in my tests? don't have a real setup at this time to test WAN >> connections, but I'd like to think that these results can be >> extrapolated to a WAN link also. > > > Uhm, in your previous test no-compression came out best. I thought you > wanted to test if sending on-disk compressed blocks over a network was a > gain. Not a plain test of compression algoritms. > > As your current numbers show: the overhead of mbuffer.log is much higher > than the overhead of compression+decompression for the lz4 case. real: > 0m29 to 1m58. (All other compressors are actually faster with mbuffer in > between? Probably because of more efficient buffering by mbuffer.) > As the overhead of mbuffer is large it is a question how much gain you > get > from removing the extra compression step by sending the on-disk > compressed > blocks. > I guess the time would go from 1m58 to about 1m30. > > NB: By splitting your zfs send/receive command in two, the numbers of > time > can be affected. Time prints the output as soon as zfs send ends. > > Have fun, > Ronald. > > > >> ________________________________________ >> From: Ronald Klop = =5Bronald-lists=40klop.ws=5D >> Sent: Tuesday, February 28, 2017 11:44 AM >> To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan >> Cc: freebsd-fs=40freebsd.org; Jeremy = Faulkner >> Subject: Re: FreeBSD restartable send/receive over WAN >> >> On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan >> > wrote: >> >>> thanks for all the pointers for the compression algorithms. I ran a >>> few >>> tests to compare compression overhead. These are local zfs >>> send/receive, and no network is involved. >>> >>> time zfs send -v =7C =7C =7C zfs >>> receive -s >>> >>> Here are the performance results that I got. >>> >>> no compression: >>> real 0m20.892s >>> user 0m0.000s >>> sys 0m5.587s >>> >>> xz -0: >>> real 8m38.569s >>> user 10m28.551s >>> sys 0m6.866s >>> >>> pxz -0: >>> real 4m38.448s >>> user 10m55.863s >>> sys 0m13.324s >>> >>> gzip: >>> real 3m51.297s >>> user 4m12.035s >>> sys 0m4.696s >>> >>> lz4: >>> real 0m29.912s >>> user 0m16.543s >>> sys 0m10.810s >>> >>> >>> lz4 has the least overhead in terms of time. pxz/xz seem to be >>> prohibitive give the above results. Unless, there is something basic >>> I'm missing? >>> >>> I was really hoping that compressed sends would be available, as that >>> would actively eliminate this overhead, given that we use lz4 as the >>> compression algorithm when writing to disks. >> >> >> Why don't you test this with a network in between? That would give much >> more valuable numbers to compare for your use case. >> The number above say nothing about the gain vs the bottleneck you are >> trying to remove. >> >> Ronald. >> >> >> >> >>> >>> >>> ________________________________ >>> From: Karli Sj=F6berg = =5Bkarli.sjoberg=40slu.se=5D >>> Sent: Sunday, February 26, 2017 8:41 AM >>> To: Gary Palmer >>> Cc: Shiva Bhanujan; Jeremy Faulkner; = freebsd-fs=40freebsd.org >>> Subject: Re: FreeBSD restartable send/receive over WAN >>> >>> >>> Den 26 feb. 2017 4:16 em skrev Gary Palmer = ::> >>>> >>>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: >>>> > The compression that we use on our ZFS filesystems is lz4. So, if I >>>> have to pipe it through a compression algorithm, that'd be >>>> uncompressing and compressing it 4 times. >>>> > >>>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) >>>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) >>>> > >>>> > isn't this quite expensive? We have to transfer multi terabyte >>>> files >>>> on a WAN link. I'm also of the understanding that gzip by itself is >>>> single-threaded, so that'd peg one of the CPUs to 100%. there might >>>> be >>>> other compression algorithms that can be used, but sending the ZFS as >>>> it is compressed on the filesystem is something that would be optimal, >>>> and would reduce the overhead of the additional =5Bde=5Dcompressions = that >>>> are taking place? >>>> >>>> Without going into the efficiency part of your message: >>>> >>>> archivers/pigz: Parallel GZIP >>>> archivers/pbzip2: Parallel BZIP2 >>>> archivers/pixz: Parallel, indexing version of XZ >>>> archivers/pxz: Parallel LZMA compressor using liblzma >>> >>> Also worth mentioning is, obviously: >>> archivers/lz4 >>> >>> :) >>> >>> /K >>> >>>> >>>> Regards, >>>> >>>> Gary >>>> >>>> > >>>> > ________________________________________ >>>> > From: = owner-freebsd-fs=40freebsd.org = =5Bowner-freebsd-fs=40freebsd.org=5D= on >>>> behalf of Jeremy Faulkner = =5Bgldisater=40gmail.com=5D >>>> > Sent: Saturday, February 25, 2017 4:03 PM >>>> > To: freebsd-fs=40freebsd.org >>>> > Subject: Re: FreeBSD restartable send/receive over WAN >>>> > >>>> > Pipe it through a compressor >>>> > >>>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: >>>> > > Hi, >>>> > > >>>> > > I just tried restartable send/receive in 10.3 and it works like a >>>> charm. I was wondering if compressed send has made its way into >>>> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the >>>> -c/--compressed option. Any pointers? >>>> > > >>>> > > Regards, >>>> > > Shiva >>>> > > >>>> > > >>>> > > ________________________________________ >>>> > > From: = owner-freebsd-fs=40freebsd.org = =5Bowner-freebsd-fs=40freebsd.org=5D >>>> on behalf of Adam Nowacki = =5Bnowakpl=40platinum.linux.pl=5D >>>> > > Sent: Thursday, February 16, 2017 10:41 AM >>>> > > To: freebsd-fs=40freebsd.org >>>> > > Subject: Re: FreeBSD restartable send/receive over WAN >>>> > > >>>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote: >>>> > >> Hello, >>>> > >> >>>> > >> I was wondering if restartable send/receive is available in >>>> FreeBSD? We're running 10.2 and have a requirement of sending and >>>> receiving ZFS snapshots over a WAN link. The snapshots could be more >>>> than a few terabytes. >>>> > >> >>>> > >> Can somebody please give me pointers, and if this feature is or >>>> isn't available in FreeBSD? >>>> > > >>>> > > FreeBSD 10.3 and later. >>>> > > >>>> > > _______________________________________________ >>>> > > freebsd-fs=40freebsd.org mailing = list >>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > > To unsubscribe, send any mail to >>>> = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>>> > > _______________________________________________ >>>> > > freebsd-fs=40freebsd.org mailing = list >>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > > To unsubscribe, send any mail to >>>> = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>>> > > >>>> > _______________________________________________ >>>> > freebsd-fs=40freebsd.org mailing = list >>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > To unsubscribe, send any mail to >>>> = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>>> > _______________________________________________ >>>> > freebsd-fs=40freebsd.org mailing = list >>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > To unsubscribe, send any mail to >>>> = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>>> > >>>> _______________________________________________ >>>> freebsd-fs=40freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to = =22freebsd-fs-unsubscribe=40freebsd.org=22 >>> _______________________________________________ >>> freebsd-fs=40freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to = =22freebsd-fs-unsubscribe=40freebsd.org=22 ________________________________