From owner-freebsd-fs@freebsd.org Fri Mar 3 04:20:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76473CF4BBF for ; Fri, 3 Mar 2017 04:20:38 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B5E21445; Fri, 3 Mar 2017 04:20:38 +0000 (UTC) (envelope-from eborisch@gmail.com) Received: by mail-io0-x230.google.com with SMTP id l7so67711736ioe.3; Thu, 02 Mar 2017 20:20:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BZB1YtrzNAQpvFLBo4K3aQQPy/aI84xgyflTXlcT9J0=; b=IlWZ0wCyoo5ZDBjtOMhDU5Qr4VsmEnnomtVXgmWEbbHpVROLA6Tjxqn2cZ4wZ8lDNK 0isd+8ZRZGhFuhL2bRj5bg5HTgbXzm8iSMszNVWVMuoHcdAiFinfvC9PJWIZ66c5tlYy Vx+hZYpDfCslsU2/G9dPYOmN9WwsRG4Bvno7ZiyDy3ErIz3uzMMwowq+0gxqQm5MqAvf M19QiQkycj+XpnA0CNo1u0IMEBLRNTELa2AoHfLb/yy9Eeml4xbXLnxVMAZ/FDlowQGb VbEAvuso/kMK+zPeptylhmaKREXFN3gA/qzWKHFWfFcq4eTj8ZQ7bnElI+3pNHbfb9TK dKSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BZB1YtrzNAQpvFLBo4K3aQQPy/aI84xgyflTXlcT9J0=; b=j9zonD2ftl6zbw3NJGaVJtUHWhIAZav5tDm1tGIsVEj+K046Lprao7biJYKGN+78hz bRFrYbsz5B8AeSWpIqdWkXabQE3T1QqEas8NJWzo9IcnTalNFf5hLA8Sbjsr45NIk4Z4 3lGyP3xegKJkr5CKpG/0RV9316d8A6OCuX4jLz5YTPLI1sy+qNrld5HSM3xLRua00oFP B1HyKQImM1SowFImeImSM7HYcXffehCMPIVNIR7aU+izs0j4nFu1mAfdw+u0s0VWeQ9u UTJdmPsS4bEvOC7x1Fygj57rOzLeR1SXNPQbtDNBJdfHbecc0M0zjlubE2mH+i5xBQSZ a2Dw== X-Gm-Message-State: AMke39kpV80MEG4MkcKXN1H/IjLSZ4R60aocmCu8KjXAQZwcuB1BsTWdyYwJ0aqMum2slc9qL0bcWtsWSJW7hQ== X-Received: by 10.107.183.20 with SMTP id h20mr1133720iof.18.1488514837530; Thu, 02 Mar 2017 20:20:37 -0800 (PST) MIME-Version: 1.0 References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> From: "Eric A. Borisch" Date: Fri, 03 Mar 2017 04:20:26 +0000 Message-ID: Subject: Re: FreeBSD restartable send/receive over WAN To: Gary Palmer , =?UTF-8?Q?Karli_Sj=C3=B6berg?= , Ronald Klop , Shiva Bhanujan Cc: Jeremy Faulkner , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Mar 2017 04:20:38 -0000 You can rate limit with mbuffer if you want experimental results for different bandwidths. (If you know what you can expect to get on your WAN, that would be a key point to test.) There are very likely crossover points in your "rankings" as a function of BW, depending on the compressibility of your data stream (and each compressor's ability to exploit it.) (Also with different machines/CPU.) Or you can measure the constituent parts to create a model and do the math across simulated link BW. You'll need throughputs and compression ratios for each compressor. Mbuffer without the -q should report the transferred (compressed) size. Decompression rate is likely faster than compression and therefore of little impact in the grand scheme. -Eric On Thu, Mar 2, 2017 at 6:35 PM Shiva Bhanujan wrote: > I ran the same set of tests between 2 FreeBSD instances, connected on a 1= G > LAN. The the comms was setup using mbuffer. Here's the basic command. > > time zfs send -v | | mbuffer -O :8099 -b 1024 -= m > 128M -P 10 -q -l /tmp/mbuffer.log > mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | = | > zfs recv > > Here are the results that I got. > > no compression: > real 3m18.591s > user 0m0.390s > sys 0m8.177s > > xz -0: > real 7m26.349s > user 7m6.436s > sys 0m8.471s > > pxz -0: > real 2m28.470s > user 6m44.168s > sys 0m12.002s > > gzip: > real 3m12.482s > user 3m8.260s > sys 0m4.974s > > lz4: > real 1m58.363s > user 0m10.000s > sys 0m8.708s > > > > lz4 still comes out best. Is there anything else that I might be missing > in my tests? don't have a real setup at this time to test WAN connection= s, > but I'd like to think that these results can be extrapolated to a WAN lin= k > also. > > > ________________________________________ > From: Ronald Klop [ronald-lists@klop.ws] > Sent: Tuesday, February 28, 2017 11:44 AM > To: Karli Sj=C3=B6berg; Gary Palmer; Shiva Bhanujan > Cc: freebsd-fs@freebsd.org; Jeremy Faulkner > Subject: Re: FreeBSD restartable send/receive over WAN > > On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan > wrote: > > > thanks for all the pointers for the compression algorithms. I ran a fe= w > > tests to compare compression overhead. These are local zfs > > send/receive, and no network is involved. > > > > time zfs send -v | | | zfs > > receive -s > > > > Here are the performance results that I got. > > > > no compression: > > real 0m20.892s > > user 0m0.000s > > sys 0m5.587s > > > > xz -0: > > real 8m38.569s > > user 10m28.551s > > sys 0m6.866s > > > > pxz -0: > > real 4m38.448s > > user 10m55.863s > > sys 0m13.324s > > > > gzip: > > real 3m51.297s > > user 4m12.035s > > sys 0m4.696s > > > > lz4: > > real 0m29.912s > > user 0m16.543s > > sys 0m10.810s > > > > > > lz4 has the least overhead in terms of time. pxz/xz seem to be > > prohibitive give the above results. Unless, there is something basic > > I'm missing? > > > > I was really hoping that compressed sends would be available, as that > > would actively eliminate this overhead, given that we use lz4 as the > > compression algorithm when writing to disks. > > > Why don't you test this with a network in between? That would give much > more valuable numbers to compare for your use case. > The number above say nothing about the gain vs the bottleneck you are > trying to remove. > > Ronald. > > > > > > > > > > ________________________________ > > From: Karli Sj=C3=B6berg [karli.sjoberg@slu.se] > > Sent: Sunday, February 26, 2017 8:41 AM > > To: Gary Palmer > > Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org > > Subject: Re: FreeBSD restartable send/receive over WAN > > > > > > Den 26 feb. 2017 4:16 em skrev Gary Palmer : > >> > >> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: > >> > The compression that we use on our ZFS filesystems is lz4. So, if I > >> have to pipe it through a compression algorithm, that'd be > >> uncompressing and compressing it 4 times. > >> > > >> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network) > >> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) > >> > > >> > isn't this quite expensive? We have to transfer multi terabyte file= s > >> on a WAN link. I'm also of the understanding that gzip by itself is > >> single-threaded, so that'd peg one of the CPUs to 100%. there might b= e > >> other compression algorithms that can be used, but sending the ZFS as > >> it is compressed on the filesystem is something that would be optimal, > >> and would reduce the overhead of the additional [de]compressions that > >> are taking place? > >> > >> Without going into the efficiency part of your message: > >> > >> archivers/pigz: Parallel GZIP > >> archivers/pbzip2: Parallel BZIP2 > >> archivers/pixz: Parallel, indexing version of XZ > >> archivers/pxz: Parallel LZMA compressor using liblzma > > > > Also worth mentioning is, obviously: > > archivers/lz4 > > > > :) > > > > /K > > > >> > >> Regards, > >> > >> Gary > >> > >> > > >> > ________________________________________ > >> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on > >> behalf of Jeremy Faulkner [gldisater@gmail.com] > >> > Sent: Saturday, February 25, 2017 4:03 PM > >> > To: freebsd-fs@freebsd.org > >> > Subject: Re: FreeBSD restartable send/receive over WAN > >> > > >> > Pipe it through a compressor > >> > > >> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: > >> > > Hi, > >> > > > >> > > I just tried restartable send/receive in 10.3 and it works like a > >> charm. I was wondering if compressed send has made its way into > >> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the > >> -c/--compressed option. Any pointers? > >> > > > >> > > Regards, > >> > > Shiva > >> > > > >> > > > >> > > ________________________________________ > >> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] > >> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl] > >> > > Sent: Thursday, February 16, 2017 10:41 AM > >> > > To: freebsd-fs@freebsd.org > >> > > Subject: Re: FreeBSD restartable send/receive over WAN > >> > > > >> > > On 2017-02-16 19:22, Shiva Bhanujan wrote: > >> > >> Hello, > >> > >> > >> > >> I was wondering if restartable send/receive is available in > >> FreeBSD? We're running 10.2 and have a requirement of sending and > >> receiving ZFS snapshots over a WAN link. The snapshots could be more > >> than a few terabytes. > >> > >> > >> > >> Can somebody please give me pointers, and if this feature is or > >> isn't available in FreeBSD? > >> > > > >> > > FreeBSD 10.3 and later. > >> > > > >> > > _______________________________________________ > >> > > freebsd-fs@freebsd.org mailing list > >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > > To unsubscribe, send any mail to > >> "freebsd-fs-unsubscribe@freebsd.org" > >> > > _______________________________________________ > >> > > freebsd-fs@freebsd.org mailing list > >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > > To unsubscribe, send any mail to > >> "freebsd-fs-unsubscribe@freebsd.org" > >> > > > >> > _______________________________________________ > >> > freebsd-fs@freebsd.org mailing list > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org= " > >> > _______________________________________________ > >> > freebsd-fs@freebsd.org mailing list > >> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org= " > >> > > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >