From owner-freebsd-fs@FreeBSD.ORG Fri Oct 3 05:13:37 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 05BEB48F for ; Fri, 3 Oct 2014 05:13:37 +0000 (UTC) Received: from mail-pa0-x231.google.com (mail-pa0-x231.google.com [IPv6:2607:f8b0:400e:c03::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6B8E686 for ; Fri, 3 Oct 2014 05:13:36 +0000 (UTC) Received: by mail-pa0-f49.google.com with SMTP id hz1so936901pad.8 for ; Thu, 02 Oct 2014 22:13:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=7ixCAG5tLo8cXRtabfO3MKyHnN2YwaCWez3lA1VMMz4=; b=jUKqrzDYFRom0kON1y0iXjTvIgDNMRaMJmKl0Llm9BgjMenjiwRX8cTPpmBhqSXxZR GB4IrY1DoSdJ+BM817LD978jHzQ5ePYZWqe8r065XdqervK1XY2o76TqVQREz+foTaRe r93FB16cij4MyEbpdKErrHO7y+sCrTMrOONZez+5BLqP65pAW8eSSa+TOgUsjK0YICLG jHg9+QE1XQfojHfSs9VjIHMIl+FrP0pikrWLsHohPJry4jzDCCWdIprr4wJ6IiPy9BHm LhBdSLyS8HaIzvnrVcoLjxS4kCybaGfx2yYGbZVhILz+Bg4cCPDesNnvBenfv6DhET+s wZtA== X-Received: by 10.67.23.13 with SMTP id hw13mr4581277pad.69.1412313216189; Thu, 02 Oct 2014 22:13:36 -0700 (PDT) Received: from macmini.internal.coolpeople.us ([2001:470:b:2d8::8001]) by mx.google.com with ESMTPSA id v11sm5598300pas.24.2014.10.02.22.13.34 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 02 Oct 2014 22:13:35 -0700 (PDT) From: John Terrell Message-Id: <4526E82D-8AEF-4A3F-B628-C26D044E331E@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. Date: Thu, 2 Oct 2014 22:13:33 -0700 References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> <99C93627-2B9C-423F-9C3D-D8B6ACB9C74A@gmail.com> <5C3F3898-9C40-48A6-9B8D-C2118448BE42@gmail.com> <2CAB03AD-B755-4F85-A6CA-E30B76517269@gmail.com> To: "freebsd-fs@freebsd.org" In-Reply-To: <2CAB03AD-B755-4F85-A6CA-E30B76517269@gmail.com> X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Oct 2014 05:13:37 -0000 Ok, I=92ve confirmed this isn=92t a true ZFS send/receive issue - it is, = in fact, an environment issue here. There=92s a switch in my = environment that a few times a day goes put to lunch (disconnects) for = 5-10 seconds. This disconnection was enough to trip up the = send/receive process but not enough to disrupt the SSH session that = initiated the transfer. =20 Thanks for all of the help and sorry to randomize the discussion. On Oct 1, 2014, at 6:57 AM, John Terrell wrote: > Eureka! I think I=92ve found the issue and it=92s nothing to do with = ZFS - it=92s an infrastructure problem. I was running another test = last night and just happened to be watching about two hours into the = test when the Mac I use to connect to the network suddenly (and = unexpectedly) lost connectivity for about five seconds. When it came = back, the SSH connection was intact but the ZFS transfer in progress = immediately failed. I=92m not sure if this is a DHCP lease = re-aquisition or what but to see if this is indeed the problem, I gave = the Mac a manual IP address and started the transfer over night. As of = now, it=92s been running for about 7 hours without issue. I=92m not = quite ready to call it solved until this transfer completes successfully = but this is the furthest it=92s gotten. >=20 >=20 > On Sep 30, 2014, at 8:02 PM, John Terrell = wrote: >=20 >> Sure. On the SmartOS (receiving) side: >>=20 >> [root@smartos /]# zpool get all zones | grep feature@ >> zones feature@async_destroy enabled = local >> zones feature@empty_bpobj active = local >> zones feature@lz4_compress active = local >> zones feature@multi_vdev_crash_dump enabled = local >> zones feature@spacemap_histogram active = local >> zones feature@enabled_txg active = local >> zones feature@hole_birth active = local >> zones feature@extensible_dataset enabled = local >> zones feature@embedded_data active = local >> zones feature@bookmarks enabled = local >> zones feature@filesystem_limits enabled = local >>=20 >> =85on the FreeBSD 10 (sending) side: >>=20 >> [root@freebsd /]# zpool get all tank | grep feature@ >> tank feature@async_destroy enabled = local >> tank feature@empty_bpobj active = local >> tank feature@lz4_compress active = local >> tank feature@multi_vdev_crash_dump enabled = local >>=20 >>=20 >> So, the FreeBSD side is a subset of the SmartOS side. >>=20 >>=20 >> On Sep 30, 2014, at 7:51 PM, Rich wrote: >>=20 >>> It's possible that your feature flags and the receiving end's aren't >>> compatible - can you show us a zfs get all tank on both sides? >>>=20 >>> - Rich >>>=20 >>> On Tue, Sep 30, 2014 at 10:39 PM, John Terrell = wrote: >>>> By the way, here's another report that sounds similar to mine = (though on Linux): >>>>=20 >>>> = http://stackoverflow.com/questions/19754375/why-does-zsh-send-fail-midway >>>>=20 >>>> His repro doesn't even use ssh (appears to be completely local). >>>>=20 >>>>=20 >>>> On Sep 30, 2014, at 6:41 PM, John Terrell = wrote: >>>>=20 >>>>> Sorry for the tardy response (busy at work. :)). Here's the = command and output (run from the SmartOS box): >>>>>=20 >>>>> ssh -c arcfour256 root@freebsdnas zfs send -R tank@daily20140920 | = zfs receive -v -F zones/tank >>>>>=20 >>>>> The output (after running for a long time xferring data): >>>>>=20 >>>>> receiving full stream of tank@daily20140920 into = zones/tank@daily20140920 >>>>> received 1.35GB stream in 15 seconds (91.9MB/sec) >>>>> receiving full stream of tank/photography@daily20140920 into = zones/tank/photography@daily20140920 >>>>> Read from remote host nas00: Connection reset by peer >>>>> cannot receive new filesystem stream: invalid backup stream >>>>>=20 >>>>>=20 >>>>> Is it possible the command is simply timing out for some reason = and closing the connection? >>>>>=20 >>>>>=20 >>>>> On Sep 26, 2014, at 3:04 PM, Will Andrews = wrote: >>>>>=20 >>>>>> What do the zfs commands print? The -v option prints some = metadata that might help diagnose the problem. >>>>>>=20 >>>>>> --Will. >>>>>>=20 >>>>>> On Monday, September 22, 2014, John Terrell = wrote: >>>>>> Hello all, I'm trying to replicate one of the ZFS filesystems = that lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. = The transfer dies at about the 1.7TB (of almost 4TB) mark indicating: >>>>>>=20 >>>>>> "cannot receive incremental stream: invalid backup stream" >>>>>>=20 >>>>>> The stream being sent is not incremental so I'm not sure what the = receiver is trying to do. Here's the command I'm using to transfer = (executed on the SmartOS machine): >>>>>>=20 >>>>>> ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive = -v tank >>>>>>=20 >>>>>> I've also tried to use mbuffer as the transport interface = (removing SSH from the equation) and it has the same result. >>>>>>=20 >>>>>> Is the possibly an incompatibility between FreeBSD10 and SmartOS = ZFS implementations? >>>>>> _______________________________________________ >>>>>> freebsd-fs@freebsd.org mailing list >>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>>>>=20 >>>>=20 >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>=20 >=20