From owner-freebsd-current@FreeBSD.ORG Fri Jul 25 09:58:56 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1DD3AB6E; Fri, 25 Jul 2014 09:58:56 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id B5A882C1D; Fri, 25 Jul 2014 09:58:55 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3hKQq157tZzRG; Fri, 25 Jul 2014 11:58:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla2; t=1406282329; x=1408874330; bh=iQG sxwTE3CG9DTK8vqPZl69YvnECUv5Skg+aJftyidM=; b=DlBaoJr/B8gAx6igCH9 S1swqgdELa8n2Xyiiam14FPGDGeyz48b0MgRsmtYpaVrvP7B5GjLcBzxKV1Vpl4f fn4cwsopo9d1B7c4hKgIA0QYAw2vAVHps6BJs9IJpT1/MGsfDFjNkXSnaWtuHof4 yO1kbChdSLB7OwpQqBiQyT+s= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id gmjTzVhItnlS; Fri, 25 Jul 2014 11:58:49 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Fri, 25 Jul 2014 11:58:48 +0200 (CEST) Received: from neli.ijs.si (neli.ijs.si [IPv6:2001:1470:ff80:88:21c:c0ff:feb1:8c91]) by mildred.ijs.si (Postfix) with ESMTP id 3hKQpw2cQ8z5W; Fri, 25 Jul 2014 11:58:48 +0200 (CEST) Received: from sleepy.ijs.si ([2001:1470:ff80:e001::1:1]) by neli.ijs.si with HTTP (HTTP/1.1 POST); Fri, 25 Jul 2014 11:58:48 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 25 Jul 2014 11:58:48 +0200 From: Mark Martinec To: Larry Rosenman Subject: Re: zfs send/recv: STILL invalid Backup Stream Organization: J. Stefan Institute In-Reply-To: <33c478c3120c21d6cb9c325cc1119cf8@thebighonker.lerctr.org> References: <62315eb454a95db636b7764aad3c0f9b@thebighonker.lerctr.org> <53D1448C.40908@freebsd.org> <53D15438.6040105@freebsd.org> <0a9cd451c3b4304d2b9d899fcb3decc3@thebighonker.lerctr.org> <3d2aac84d962a703fbf56a864ba5f19c@mailbox.ijs.si> <9eef3e7e964cb33fd163cc2f9300f326@mailbox.ijs.si> <53D1AB3D.1060205@freebsd.org> <33c478c3120c21d6cb9c325cc1119cf8@thebighonker.lerctr.org> Message-ID: <60845088153247fa31c50c9f52ef72cb@mailbox.ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.0.1 Cc: Freebsd fs , freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 09:58:56 -0000 > On 2014-07-24 19:56, Allan Jude wrote: >>> or better yet: >>> ssh root@tbh.lerctr.org "zfs send ..." | mbuffer -m 16M | zfs recv >>> ... >>> (The misc/mbuffer compensates for bursty zfs reads and writes. >>> A note to myself: I should suggest to Allan to add mbuffer >>> in a pipe as used in sysutils/zxfer, instead of patching zxfer >>> for our local use :) >> >> zxfer can already do this, with the -D option >> I actually use misc/clpbar and get a progress bar as well >> >> -D 'bar -s %%size%% -bl 1m -bs 128m' >> >> or in your case: -D 'mbuffer -m 16M' Thank you Allan! It shows it's been a while since the last time I inspected the guts of zxfer. 2014-07-25 06:43 Larry Rosenman wrote: > Ok, I did the mbuffer trick, and the SEND side is where the memory > issue is: > borg.lerctr.org /home/ler/bin $ tail zfs-send.log > 23:28:12 15.7G zroot/home@2014-07-24_22:56 > 23:28:13 15.7G zroot/home@2014-07-24_22:56 > 23:28:14 15.7G zroot/home@2014-07-24_22:56 > 23:28:15 15.7G zroot/home@2014-07-24_22:56 > 23:28:16 15.7G zroot/home@2014-07-24_22:56 > 23:28:17 15.7G zroot/home@2014-07-24_22:56 > 23:28:18 15.7G zroot/home@2014-07-24_22:56 > 23:28:19 15.7G zroot/home@2014-07-24_22:56 > 23:28:20 15.8G zroot/home@2014-07-24_22:56 > Write failed: Cannot allocate memory > borg.lerctr.org /home/ler/bin $ Good information. So the "Write failed: Cannot allocate memory" on the send side is what is causing the "invalid backup stream" on the receiving side. > borg.lerctr.org /home/ler/bin $ tail zfs-recv.log > cannot receive new filesystem stream: invalid backup stream > borg.lerctr.org /home/ler/bin $ > > borg.lerctr.org /home/ler/bin $ cat backup-TBH-ZFS-initial.sh > #!/bin/sh > DATE=`date "+%Y-%m-%d_%H:%M"` > #DATE2=2013-03-24 > #DATE2=`date -v "-1d" "+%Y-%m-%d"` > # snap the source > ssh root@tbh.lerctr.org zfs snapshot -r zroot@${DATE} > # zfs copy the source to here. > ssh root@tbh.lerctr.org 2>zfs-send.log "zfs send -v -R zroot@${DATE} " > | \ > mbuffer -m 16M 2>mbuffer.log | \ > zfs recv -F -u -v -d zroot/backups/TBH3 2>zfs-recv.log > # make sure we NEVER allow the backup stuff to automount. > /sbin/zfs list -H -t filesystem -r zroot/backups/TBH3| \ > awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh > borg.lerctr.org /home/ler/bin $ > > borg.lerctr.org /home/ler/bin $ ssh tbh vmstat -z > ITEM SIZE LIMIT USED FREE REQ FAIL > SLEEP [...] > Where to now? Don't know, I'd guess some network-related memory limit is being hit on the sending site. Why not try to decouple the 'zfs send' from a network copy and ssh: Login to a remote side, do a 'zfs send' to some local temporary file there, then feed that file to ssh and send it over to a home host, where it can be piped into some simple program like md5 or 'wc -c' instead of a zfs recv. Mark