Date: Wed, 06 Mar 2013 04:49:48 -0600 From: Larry Rosenman <ler@lerctr.org> To: <freebsd-stable@freebsd.org>, <freebsd-current@freebsd.org> Subject: Fwd: Re: zfs send/recv invalid data Message-ID: <46d966dd574cd8097d4972213c73e9be@webmail.lerctr.org>
next in thread | raw e-mail | index | archive | help
I forgot to add current/stable to the list
TL;DR: there seems(!) to be something(!) unclean about an ssh path
between an 8.3-STABLE(r247820) and
10.0-CURRENT(r247826) box such that a zfs send stream is corrupted in
transit.
below is the thread from -fs about it, with sshd configs from both
sides.
If I copy the stream it works, but piping through ssh does NOT.
-------- Original Message --------
Subject: Re: zfs send/recv invalid data
Date: 2013-03-06 04:46
From: Larry Rosenman <ler@lerctr.org>
To: Steven Hartland <killing@multiplay.co.uk>
Cc: Ronald Klop <ronald-freebsd8@klop.yi.org>, <freebsd-fs@freebsd.org>
On 2013-03-06 02:38, Steven Hartland wrote:
> ----- Original Message ----- From: "Larry Rosenman" <ler@lerctr.org>
>>>>>>>> I received an "invalid data" in a zfs send (from 8.3) / zfs
>>>>>>>> recv (to 10.0) of a -R -I stream.
>>>>>>>> What data do I need to gather to figure out what side and
>>>>>>>> what's wrong?
>>>>>>>> I've already started zpool scrubs on both sides.
>>>>>>>> I can insert a tee to grab the stream on either/both sides if
>>>>>>>> that would help.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Is the problem repeatable or is it just a network glitch?
>>>>>>> Ronald.
>>>>>> Repeatable.......
>>>>> Here is the exact error message:
>>>>> receiving incremental stream of vault/home/ctr@2013-03-05-test3
>>>>> into zroot/backups/TBH/home/ctr@2013-03-05-test3
>>>>> cannot receive incremental stream: invalid backup stream
>>>>> this is the script I'm running:
>>>>> #!/bin/sh
>>>>> DATE=`date "+%Y-%m-%d-BUG-REPRO"`
>>>>> DATE2=`date -v "-1d" "+%Y-%m-%d"`
>>>>> # snap the source
>>>>> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE}
>>>>> # zfs copy the source to here.
>>>>> ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2}
>>>>> vault@${DATE} | \
>>>>> tee /tmp/backup.stream.send.${DATE} | \
>>>>> ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE}
>>>>> | zfs recv -u -v -d zroot/backups/TBH\""
>>>>> # make sure we NEVER allow the backup stuff to automount.
>>>>> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \
>>>>> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
>>>>> both streams are in http://www.lerctr.org/~ler/ZFS_RECV
>>>> Your send and receive sides differ, which indicates your ssh
>>>> shell my not be clean.
>>>> Looking at the receive side its got what looks like a mail
>>>> message appended.
>>>> I suspect if you manually copy the receive copy to the 10 machine
>>>> and
>>>> the receive it will work fine.
>>> we're copying mail files........
>>> and it still fails....
>>>
>> I've put more example send/recv files in that directory.
>> we're copying home dirs, which include lots of mail.
>> (this one is my wife's)
>> Ideas?
>> I *CAN* give access to both sides via ssh.....
> The copy of the data stream on both sides should be identical
> though and its not, which leads me to believe something is
> corrupting the data on the way. Try the following:-
>
>> From source:-
> zfs send -R -D -I vault@${DATE2} vault@${DATE} > test.stream
> scp test.stream home.lerctr.org:~/
>> From target:
> zfs recv -u -v -d zroot/backups/TBH < test.stream
> If this works then there is something unclean about your ssh
> shell.
> Regards
> Steve
> send side:
# zfs send -R -D -I vault@2013-03-05 vault@2013-03-06 >/tmp/send.stream
# openssl md5 /tmp/send.stream
MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33
# scp /tmp/send.stream home:/tmp/send.stream
send.stream 100% 1180MB 2.5MB/s
07:44
# uname -a
FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54
r247820: Mon Mar 4 18:08:11 CST 2013
root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64
#
Receive side:
# uname -a
FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826:
Mon Mar 4 19:59:08 CST 2013
root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64
# openssl md5 /tmp/send.stream
MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33
# zfs recv -F -u -v -d zroot/backups/TBH < /tmp/send.stream
<success>
#
So, you are correct that something(tm) is unclean about the ssh path.
adding -current and -stable for diagnosing ssh issue.
sshd config on the 8.3-STABLE box:
# cat /etc/ssh/sshd_config
# $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $
# $FreeBSD: stable/8/crypto/openssh/sshd_config 247521 2013-03-01
02:06:04Z des $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
# Note that some of FreeBSD's defaults differ from OpenBSD's, and
# FreeBSD has a few additional options.
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
# Disable legacy (protocol version 1) support in the server for new
# installations. In future the default will change to require explicit
# activation of protocol 1
Protocol 2
# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024
# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#RSAAuthentication yes
#PubkeyAuthentication yes
# The default is to check both .ssh/authorized_keys and
.ssh/authorized_keys2
# but this is overridden so installations will only check
.ssh/authorized_keys
#AuthorizedKeysFile .ssh/authorized_keys
#AuthorizedPrincipalsFile none
# For this to work you will also need host keys in
/etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# Change to yes to enable built-in password authentication.
#PasswordAuthentication no
#PermitEmptyPasswords no
# Change to no to disable PAM authentication
#ChallengeResponseAuthentication yes
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
# Set this to 'no' to disable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM yes
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
ClientAliveInterval 120
ClientAliveCountMax 200000
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum FreeBSD-20120901
# no default banner path
#Banner none
# override default of no subsystems
Subsystem sftp /usr/libexec/sftp-server
# Disable HPN tuning improvements.
#HPNDisabled no
# Buffer size for HPN to non-HPN connections.
#HPNBufferSize 2048
# TCP receive socket buffer polling for HPN. Disable on non autotuning
kernels.
#TcpRcvBufPoll yes
# Allow the use of the NONE cipher.
#NoneEnabled no
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
#
sshd config on the 10.0-CURRENT:
# cat /etc/ssh/sshd_config
# $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $
# $FreeBSD: head/crypto/openssh/sshd_config 240075 2012-09-03
16:51:41Z des $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
# Note that some of FreeBSD's defaults differ from OpenBSD's, and
# FreeBSD has a few additional options.
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
# The default requires explicit activation of protocol 1
#Protocol 2
# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024
# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#RSAAuthentication yes
#PubkeyAuthentication yes
# The default is to check both .ssh/authorized_keys and
.ssh/authorized_keys2
# but this is overridden so installations will only check
.ssh/authorized_keys
AuthorizedKeysFile .ssh/authorized_keys
#AuthorizedPrincipalsFile none
# For this to work you will also need host keys in
/etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# Change to yes to enable built-in password authentication.
#PasswordAuthentication no
#PermitEmptyPasswords no
# Change to no to disable PAM authentication
#ChallengeResponseAuthentication yes
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
# Set this to 'no' to disable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM yes
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
ClientAliveInterval 120
ClientAliveCountMax 200000
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum FreeBSD-20120901
# no default banner path
#Banner none
# override default of no subsystems
Subsystem sftp /usr/libexec/sftp-server
# Disable HPN tuning improvements.
#HPNDisabled no
# Buffer size for HPN to non-HPN connections.
#HPNBufferSize 2048
# TCP receive socket buffer polling for HPN. Disable on non autotuning
kernels.
#TcpRcvBufPoll yes
# Allow the use of the NONE cipher.
#NoneEnabled no
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
#
Ideas from the ssh folks?
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46d966dd574cd8097d4972213c73e9be>