From owner-freebsd-fs@freebsd.org Thu May 13 15:50:15 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6CCE96461A9 for ; Thu, 13 May 2021 15:50:15 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fgx1y3htfz3M10; Thu, 13 May 2021 15:50:14 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:80af:9f4f:a874:2b2a] ([IPv6:2607:f3e0:0:4:80af:9f4f:a874:2b2a]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 14DFoD7R099764 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Thu, 13 May 2021 11:50:14 -0400 (EDT) (envelope-from mike@sentex.net) To: Alan Somers Cc: freebsd-fs References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> From: mike tancsa Subject: Re: speeding up zfs send | recv Message-ID: Date: Thu, 13 May 2021 11:50:14 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-Rspamd-Queue-Id: 4Fgx1y3htfz3M10 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-2.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[mike]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f3e0:0:3::19:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; SPAMHAUS_ZRD(0.00)[2607:f3e0:0:3::19:from:127.0.2.255]; DMARC_NA(0.00)[sentex.net]; TO_DN_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 May 2021 15:50:15 -0000 On 5/13/2021 11:37 AM, Alan Somers wrote: > On Thu, May 13, 2021 at 8:45 AM mike tancsa > wrote: > > For offsite storage, I have been doing a zfs send across a 10G > link and > > Why would the mail spool send be so slow compared to the sends wher= e > datasets only have a few large files ? > > > Is this a high latency link?=C2=A0 ZFS send streams can be bursty.=C2=A0= Piping > the stream through mbuffer helps with that.=C2=A0 Just google "zfs send= > mbuffer" for some examples.=C2=A0 And be aware that your speed may be > limited by the sender.=C2=A0 Especially if those small files are random= ly > spread across the platter, your sending server's disks may be the > limiting factor.=C2=A0 Use gstat to check. > -Alan Thanks for the mbuffer suggestion, I will give it a try!=C2=A0 The fiber = is just over to the next building and connected via layer 2 switch so very low latency. 5 packets transmitted, 5 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 0.042/0.057/0.087/0.015 ms zfs is all "black box" to me, but I don't understand why the contents of the dataset would make a difference ?=C2=A0 I am sending from my backup server to my offsite backup server. i.e. the mail server sends its incremental snaphots to the backup server. I then once a week focus on the latest snapshot on the backup server and send it to my offsite server.=C2=A0 Would not that zfs send just be sending blocks of data from= the zfs dataset ? and wouldn't all contents have an equal chance of being spread across the platters on the backup server ? =C2=A0=C2=A0=C2=A0 ---Mike