From owner-freebsd-fs@FreeBSD.ORG Thu Jun 13 16:06:59 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A829363F for ; Thu, 13 Jun 2013 16:06:59 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 65AC115C9 for ; Thu, 13 Jun 2013 16:06:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=2EbPCYEag64lSc/JF9mqcvzjg6mzkpzwh5L54Q7twlE=; b=KcINUEWUVyo2/TTIyDOUkauzakrfzx+Q8YxwZdEeByIbrORQFn4Dqnle7jabo7quf6c3kptLoABGaoaO5ixrEEc6sGrdIopR+8dRZG5+Yt9R9w7YWFOiibWuPtdyhtW0at0AIlp/3Z8uMfgCVqpHekqLFoCBYvhACzJBtudXHAA=; Received: from localhost.lerctr.org ([127.0.0.1]:36432 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UnA37-0000ep-Th for freebsd-fs@freebsd.org; Thu, 13 Jun 2013 11:06:59 -0500 Received: from [32.97.110.58] by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Thu, 13 Jun 2013 11:06:57 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 13 Jun 2013 11:06:57 -0500 From: Larry Rosenman To: freebsd-fs@freebsd.org Subject: Re: zfs send/recv dies when transferring large-ish dataset In-Reply-To: References: <57e0551229684b69bc27476b8a08fb91@DB3PR07MB059.eurprd07.prod.outlook.com> Message-ID: <7dbb4a3d84381d923e22ec5ed77ea15e@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.9.1 X-Spam-Score: -3.3 (---) X-LERCTR-Spam-Score: -3.3 (---) X-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.392 X-LERCTR-Spam-Report: SpamScore (-3.3/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.392 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2013 16:06:59 -0000 This may be related to the grief I'm seeing in general with send/recv. I try to send streams and it breaks with invalid datastream or similar to yours. I've posted many times, and offered up my machine(s) to debug but no takers. On 2013-06-13 10:56, Jona Schuman wrote: > machine2# nc -d -l 9999 | zfs receive -v -F -d storagepool > machine1# zfs send -v -R dataset@snap | nc machine2 9999 > > machine1-output: sending from @ to dataset@snap > machine2-output: receiving full stream of dataset@snap into > storagepool/dataset@snap > machine1-output: warning: cannot send 'dataset@snap': Broken pipe > machine1-output: Broken pipe > > > On Thu, Jun 13, 2013 at 3:42 AM, Ivailo Tanusheff > wrote: >> Hi, >> >> Can you try send/recv with the -v or with -vP swiches, so you can see >> more verbose information? >> >> Regards, >> Ivailo Tanusheff >> >> -----Original Message----- >> From: owner-freebsd-fs@freebsd.org >> [mailto:owner-freebsd-fs@freebsd.org] On Behalf Of Jona Schuman >> Sent: Thursday, June 13, 2013 2:41 AM >> To: freebsd-fs@freebsd.org >> Subject: zfs send/recv dies when transferring large-ish dataset >> >> Hi, >> >> I'm getting some strange behavior from zfs send/recv and I'm hoping >> someone may be able to provide some insight. I have two identical >> machines running 9.0-RELEASE-p3, each having a ZFS pool (zfs 5, zpool >> 28) for storage. I want to use zfs send/recv for replication between >> the two machines. For the most part, this has worked as expected. >> However, send/recv fails when transferring the largest dataset (both >> in actual size and in terms of number of files) on either machine. >> With these datasets, issuing: >> >> machine2# nc -d -l 9999 | zfs recv -d storagepool machine1# zfs send >> dataset@snap | nc machine2 9999 >> >> terminates early on the sending side without any error messages. The >> receiving end continues on as expected, cleaning up the partial data >> received so far and reverting to its initial state. (I've tried using >> mbuffer instead of nc, or just using ssh, both with similar results.) >> Oddly, zfs send dies slightly differently depending on how the two >> machines are connected. When connected through the racktop switch, zfs >> send dies quietly without any indication that the transfer has failed. >> When connected directly using a crossover cable, zfs send dies quietly >> and machine1 becomes unresponsive (no network, no keyboard, hard reset >> required). In both cases, no messages are printed to screen or to >> anything in /var/log/. >> >> >> I can transfer the same datasets successfully if I send/recv to/from >> file: >> >> machine1# zfs send dataset@snap > /tmp/dump machine1# scp /tmp/dump >> machine2:/tmp/dump machine2# zfs recv -d storagepool < /tmp/dump >> >> so I don't think the datasets themselves are the issue. I've also >> successfully tried send/recv over the network using different network >> interfaces (10GbE ixgbe cards instead of the 1GbE igb links), which >> would suggest the issue is with the 1GbE links. >> >> Might there be some buffering parameter that I'm neglecting to tune, >> which is essential on the 1GbE links but may be less important on the >> faster links? Are there any known issues with the igb driver that >> might be the culprit here? Any other suggestions? >> >> Thanks, >> Jona >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> >> > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893