From nobody Mon Jul 4 13:06:43 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4AD9E8AFC2B for ; Mon, 4 Jul 2022 13:07:00 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lc5g72fS8z579Y for ; Mon, 4 Jul 2022 13:06:59 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-qt1-f174.google.com with SMTP id g14so9781676qto.9 for ; Mon, 04 Jul 2022 06:06:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WhNttAtyT2M7hUz7s4HYygnqyCOEkEB7dEWuHj5hFdc=; b=e5p3I2kc2upVww3i107UVen6oLdcK7TQQw7GbYIRdwm1ro+i2Gxrd6GcsKz0pLzTeS 3L1imZjOuMFD4vpMxZ9Usz233ajbWFOHu7MOn0fI+xWr8p/zg3w3Vrnd9HwIYPmUZ1Eq eC1FTI/4xNkHlyDO2FOJ+p15W0pV8+hWmIrfvw+hpg58tZPEH6nenJ3uxnod/8VvYD9C krXdNveqBLjAG1MxoWIyt+nTTLYG/EfzaCYOfg35CqrYOd6cLvwb57UzTx5HxJ5fdkdH XdqykdrSJ+0QlnZ+KNC0sHInLT/tPSbLUxFfJY4ZGcySrZc+sM8AI8DhVbjzXrsTNems 0xCg== X-Gm-Message-State: AJIora9vbWIbX9KJhfoLa51uXe1bHnze7wGS2YMhMH0cUfFKScMkl+Fy zwMa7UIN512rEgYbx834MFsVmoxiLCFDXtfWq6CxlddBwhM= X-Google-Smtp-Source: AGRyM1vfzzN4rpJ7jQIRdN+STXrRt2+b3Y4Dtz8/nmkuzjSMbFXRqXBglvCZpFv+wy9CoCPAeonYpTt2fIqvQ1/AP1U= X-Received: by 2002:a05:622a:14d2:b0:31d:3a01:3db2 with SMTP id u18-20020a05622a14d200b0031d3a013db2mr13046164qtx.575.1656940013425; Mon, 04 Jul 2022 06:06:53 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Mon, 4 Jul 2022 07:06:43 -0600 Message-ID: Subject: Re: Problem ZFS send / receive 13.1-RELEASE To: thilo jeremias Cc: freebsd-stable Content-Type: multipart/alternative; boundary="000000000000ea1ac805e2fa6894" X-Rspamd-Queue-Id: 4Lc5g72fS8z579Y X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.160.174 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-2.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.981]; FREEFALL_USER(0.00)[asomers]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[209.85.160.174:from]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.160.174:from]; MLMMJ_DEST(0.00)[freebsd-stable]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --000000000000ea1ac805e2fa6894 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable It's not a bug. The problem is that you can't combine zfs send -R and also zfs recv -s . The former creates several independent send streams. But the latter generates a token that will only resume a single stream. And there's no way to fix the problem given ZFS's design. If the send gets interrupted, you need to go check on the destination side to see which file datasets are incomplete and resume all of them. On Mon, Jul 4, 2022, 5:53 AM thilo jeremias wrote: > Hello everyone, > > I posted a question on ( > https://forums.freebsd.org/threads/zfs-replication-interrupted-and-resume= d-missing-snapshots.85666/ > ) > Which I now think is a bug in zfs send/receive for 13.1-RELEASE > > > Essentially, an interrupted & restarted receive does not create all > snapshots from the stream. > > The following: > > #Interrupted transfer > zfs send -R "$D@final" | dd bs=3D10m count=3D200 | zfs recv -s "$D"_bad > > #Resume transfer > TOKEN=3D"$(zfs get -H receive_resume_token "$D"_bad | cut -f 3)" > zfs send -t "$TOKEN" | zfs recv -s "$D"_bad > > > #Good transfer > zfs send -R "$D@final" | zfs recv -s "$D"_good > > > > Produces: > > NAME USED AVAIL REFER MOUNTPOINT > Backup/testset_good@initial 85.2K - 128K - > Backup/testset_good@add100m 85.2K - 100M - > Backup/testset_good@add300m 85.2K - 400M - > Backup/testset_good@del100m 85.2K - 300M - > Backup/testset_good@del300m 0B - 128K - > Backup/testset_good@final 0B - 128K - > NAME USED AVAIL REFER MOUNTPOINT > Backup/testset_bad@initial 85.2K - 128K - > Backup/testset_bad@add100m 0B - 100M - > > > Which is not correct since it misses the snapshots from the original > stream (=E2=80=9Cgood=E2=80=9D) > > Can this be reproduced by anyone else? > > > Thilo > > > ------------------------------- > Dipl. Ing. Thilo Jeremias > Hauptstra=C3=9Fe 11 > 35466 Rabenau > > T: +49 15782492240 > > > --000000000000ea1ac805e2fa6894 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
It's not a bug. The problem is that you can't com= bine zfs send -R and also zfs recv -s .=C2=A0 The former creates several in= dependent send streams.=C2=A0 But the latter generates a token that will on= ly resume a single stream. And there's no way to fix the problem given = ZFS's design. If the send gets interrupted,=C2=A0 you need to go check = on the destination side to see which file datasets are incomplete and resum= e all of them.=C2=A0

On Mon, Jul 4, 2022, 5:53 AM thilo jeremias <thilo@nispuk.com> wrote:
Hello everyone,

Which I now think is a bug in= zfs send/receive for 13.1-RELEASE


= Essentially, an interrupted & restarted receive does not create all sna= pshots from the stream.

The following:
<= pre dir=3D"ltr" style=3D"box-sizing:border-box;font-family:Monaco,Menlo,Con= solas,"Roboto Mono","Andale Mono","Ubuntu Mono&quo= t;,monospace;font-size:13px;word-wrap:normal;margin-top:0px;margin-bottom:0= px;padding:10px 10px 0px;line-height:1.4;overflow:auto;direction:ltr;color:= rgb(20,20,20)">#Interrupted transfer=20 zfs send -R "$D@final" | dd bs=3D10m count=3D200 | zfs recv -s &q= uot;$D"_bad #Resume transfer TOKEN=3D"$(zfs get -H receive_resume_token "$D"_bad | cut= -f 3)" zfs send -t "$TOKEN" | zfs recv -s "$D"_bad 

#Good transfer
zfs send -R "$D@final" | zfs recv -s "$D"_good


Produces:= 
NAME                          USED  AVAIL     R=
EFER  MOUNTPOINT
Backup/testset_good@initial  85.2K      -      128K  -
Backup/testset_good@add100m  85.2K      -      100M  -
Backup/testset_good@add300m  85.2K      -      400M  -
Backup/testset_good@del100m  85.2K      -      300M  -
Backup/testset_good@del300m     0B      -      128K  -
Backup/testset_good@final       0B      -      128K  -
NAME                         USED  AVAIL     REFER  MOUNTPOINT
Backup/testset_bad@initial  85.2K      -      128K  -
Backup/testset_bad@add100m     0B      -      100M  -

=
Which is not correct since it misses the snapshots from the orig= inal stream (=E2=80=9Cgood=E2=80=9D)

Can this be r= eproduced by anyone else?


Thilo


-------------------------------
Dipl. Ing. Thilo Jeremias
= Hauptstra=C3=9Fe 11
35466 Rabenau

T: +49 15782492240


--000000000000ea1ac805e2fa6894--