From owner-freebsd-fs@FreeBSD.ORG Wed Mar 6 22:20:36 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6E99231A for ; Wed, 6 Mar 2013 22:20:36 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 26EB9C1A for ; Wed, 6 Mar 2013 22:20:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Content-Type:MIME-Version:References:Message-ID:In-Reply-To:Subject:cc:To:From:Date; bh=j9Iqh8w1B26ImaL/fpyK227Vcr0JdCCEZSsTj3AD0CM=; b=OZtEeCHictLvRkU4/+cTQMVUNgrUXV7ff8J5F/hMGdvpDoAiFT9SWjsibpmnIW+7np+T4hF6XTnX0ouoeM9rt4P1xkCWdjWrTD2Sy/58WTNmEl/+SpfXx35LznCd6mRdEcm677JYwPKI7XUqaaQ/53cxV3WFqS0MFGBD8gPDH80=; Received: from lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net ([2001:470:1f0e:3ad::2]:39595) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDMhI-000O16-26; Wed, 06 Mar 2013 16:20:29 -0600 Date: Wed, 6 Mar 2013 16:20:24 -0600 (CST) From: Larry Rosenman To: Ronald Klop Subject: Re: zfs send/recv invalid data In-Reply-To: Message-ID: References: <7c93aef20a88cdbcca85739e67470dce@webmail.lerctr.org> <25894116c93a59dab1fd976b425c36d1@webmail.lerctr.org> <07b59d5d4b2a3dab1385b054eea4f2da@webmail.lerctr.org> <7619c6383449c7a316edb1cdffc98c54@webmail.lerctr.org> <473BCD0A4A7A4E6AB08409A9B0C82363@multiplay.co.uk> <6dcfb2284551025af3cf58703a2b5cdc@webmail.lerctr.org> <920990505611cd96a075c80d06691bb0@webmail.lerctr.org> <201303061857.r26IvLnc024186@higson.cam.lispworks.com> <9e133e088f6c3c3dafdc0a99eb7c48c1@webmail.lerctr.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: -2.9 (--) X-LERCTR-Spam-Score: -2.9 (--) X-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1,BAYES_00=-1.9 X-LERCTR-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1,BAYES_00=-1.9 Cc: tevans.uk@googlemail.com, freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 22:20:36 -0000 On Wed, 6 Mar 2013, Ronald Klop wrote: > On Wed, 06 Mar 2013 20:15:02 +0100, Larry Rosenman wrote: > >> On 2013-03-06 12:59, Larry Rosenman wrote: >>> On 2013-03-06 12:57, Martin Simmons wrote: >>>>>>>>> On Wed, 06 Mar 2013 05:07:28 -0600, Larry Rosenman said: >>>>> On 2013-03-06 04:56, Larry Rosenman wrote: >>>>>> On 2013-03-06 04:51, Tom Evans wrote: >>>>>>> On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman >>>>>>> wrote: >>>>>>>> So, you are correct that something(tm) is unclean about the ssh >>>>>>>> path. >>>>>>>> >>>>>>> And what is it that is added? Surely that is trivial to determine? >>>>>>> echo hello > foo >>>>>>> cat foo | ssh host cat >>>>>>> Cheers >>>>>>> Tom >>>>>> it seems to be bytestream dependent, as this simple case works, and >>>>>> the vast majority of the sends work, but the particular >>>>>> case in the thread does NOT. >>>>>> # echo hello >foo >>>>>> # cat foo | ssh home cat >>>>>> hello >>>>>> # >>>>> stranger: >>>>> $ cd /tmp >>>>> $ cat send.stream | ssh home openssl md5 >>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>>> $ openssl md5 send.stream >>>>> MD5(send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 >>>>> $ >>>>> so I'm not sure what is tripping it in the stream to a pipe case. >>>> Is that running under the same uid? The script you posted earlier runs >>>> the >>>> pipe under ssh root@tbh.lerctr.org. >>>> __Martin >>> Ooo, good point. Would csh vs. sh make a difference? >> Doesn't seem to make a difference.... >> >> # cat send.stream | ssh root@home openssl md5 >> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >> # uname -a >> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 r247820: >> Mon Mar 4 18:08:11 CST 2013 >> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 >> # >> # exec csh >> root@borg:/home/ler # pwd >> /home/ler >> root@borg:/home/ler # ssh tbh "cat >> /home/ler/public_html/ZFS_RECV/send.stream | ssh root@home.lerctr.org >> openssl md5" >> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33 >> root@borg:/home/ler # uname -a >> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: Mon >> Mar 4 19:59:08 CST 2013 >> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >> root@borg:/home/ler # >> > > > Hi, > > My script to send/rcv locally has a commented mbuffer in it because I had > trouble with it in the past. > > zfs send $SEND_ARGS $VOL_FROM@$PREFIX$NEW \ > | zfs receive -F $VOL_TO > # | mbuffer -q | zfs receive -F $VOL_TO > > > Maybe zfs receive expects a certain size of blocks of data and |ssh (or in my > case |mbuffer) changes that. This is all a wild guess. More debugging should > tell the difference, but it might give a direction to look for. > > > The error "invalid backup stream" can be given at a couple of places. Can you > use a debugger or ktrace or something else to get more information about what > is happening? > > $ 20:32:24 ronald@sjakie > [/usr/src/cddl/contrib/opensolaris/lib/libzfs/common] > grep EZFS_BADSTREAM * > libzfs.h: EZFS_BADSTREAM, /* bad backup stream */ > libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, > dgettext(TEXT_DOMAIN, > libzfs_sendrecv.c: error = zfs_error(hdl, > EZFS_BADSTREAM, errbuf); > libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: error = zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: return (zfs_error(hdl, > EZFS_BADSTREAM, errbuf)); > libzfs_sendrecv.c: return (zfs_error(hdl, > EZFS_BADSTREAM, errbuf)); > libzfs_sendrecv.c: return (zfs_error(hdl, > EZFS_BADSTREAM, errbuf)); > libzfs_sendrecv.c: (void) zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: (void) zfs_error(hdl, EZFS_BADSTREAM, > errbuf); > libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, > errbuf)); > libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, > errbuf)); > libzfs_sendrecv.c: return (zfs_error(hdl, EZFS_BADSTREAM, > errbuf)); > libzfs_util.c: case EZFS_BADSTREAM: Hrm. Anyone have an idea on what to do with ktrace or dtrace or gdb or whatever to debug this? I'm at a loss on where to start on it :( -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893