From owner-freebsd-stable@freebsd.org Wed May 27 22:29:11 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A654B2F17DD for ; Wed, 27 May 2020 22:29:11 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 49XQVG56yLz4GjY for ; Wed, 27 May 2020 22:29:10 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id 04RMT12W091826 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 27 May 2020 22:29:01 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id 04RMSu6D012722 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Thu, 28 May 2020 05:28:56 +0700 (+07) (envelope-from eugen@grosbein.net) From: Eugene Grosbein Subject: Re: zfs receive -s: transfer got interrupted, but no token on the receiving side. To: freebsd-stable@freebsd.org References: Message-ID: Date: Thu, 28 May 2020 05:28:40 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 49XQVG56yLz4GjY X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [1.44 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.52)[0.516]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; NEURAL_SPAM_MEDIUM(0.48)[0.476]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[grosbein.net]; R_SPF_PERMFAIL(0.00)[empty SPF record]; NEURAL_SPAM_LONG(0.55)[0.552]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 May 2020 22:29:11 -0000 27.05.2020 22:11, Eugene M. Zheganin wrote: > Hello, > > > I have a ZFS dataset about 10T of actual size (may be more) that I need to send over a very laggy connection. So I'm sending it from the shell-script that reattempts to send it after a short timeout, retrieving the send token first. Like that: > > ===Cut=== > > #!/bin/sh > > exitstatus=1 > token=`ssh user@server zfs get receive_resume_token data/reference | grep -v SOURCE | awk '{print $3}'` > while ([ $exitstatus -ne 0 ]) > do > zfs send -t $token | ssh -C user@server sudo zfs receive -Fus data/reference > exitstatus=$? > echo "Send interrupted/ended, sleeping for 5 secs." > sleep 5 > done > > ===Cut=== > > Usually this goes just flawlessly. But this time, due to a low transfer speed and a very laggy connectivity, thus resulting in a send time of several weeks, I got a problem: > > about one reattempt out of 200 fails with a situation when there's no snapshot on the receiving side (only an incomplete dataset which is definitely smaller than original one), thus meaning that the dataset is incomplete, but there's no token on this dataset. This happened already twice, each time on a different size. > > How can this even be possible ? > > > Recieving side side: > > [root@playkey-nas:~]# zfs list -t all > NAME USED AVAIL REFER MOUNTPOINT > data 4,67T 20,8T 128K /data > data/reference 4,67T 20,8T 128K /data/reference > > (as you can see there's no snapshot) > > Sending side and the snapshot: > > [root@san1:/usr/src]# zfs list -t all | grep data/reference@ver2_5917 > data/reference@ver2_5917 44,3G - 7,73T - zfs(8) manual page tells: receive_resume_token For filesystems or volumes which have saved partially-completed state from zfs receive -s, this opaque token can be provided to zfs send -t to resume and complete the zfs receive. Note that /bin/sh does NOT support storing arbitrary opaque data in its variables in unencoded form. Maybe sometimes you get token with characters that break such code. Here is how I encode completely opaque and even binary data when I need to store in a shell variable: key=$(command_printing_data | b64encode -r -) # use binary key later: echo "$key" | b64decode -r | geli attach -pk - $p This technique works for any kind of data including special symbols, zero bytes etc.