Date: Fri, 25 Jul 2014 11:58:48 +0200 From: Mark Martinec <Mark.Martinec+freebsd@ijs.si> To: Larry Rosenman <ler@lerctr.org> Cc: Freebsd fs <freebsd-fs@freebsd.org>, freebsd-current@freebsd.org Subject: Re: zfs send/recv: STILL invalid Backup Stream Message-ID: <60845088153247fa31c50c9f52ef72cb@mailbox.ijs.si> In-Reply-To: <33c478c3120c21d6cb9c325cc1119cf8@thebighonker.lerctr.org> References: <62315eb454a95db636b7764aad3c0f9b@thebighonker.lerctr.org> <53D1448C.40908@freebsd.org> <d0612ef9399b4e95bd09e5271c91505b@thebighonker.lerctr.org> <eefa10e173e2e23a88c85c38221a1c22@thebighonker.lerctr.org> <53D15438.6040105@freebsd.org> <0a9cd451c3b4304d2b9d899fcb3decc3@thebighonker.lerctr.org> <cc635a377e817cb57d27650ed3e558f4@thebighonker.lerctr.org> <3d2aac84d962a703fbf56a864ba5f19c@mailbox.ijs.si> <c93ef94a8e9e076b86a65f4986d0e30d@thebighonker.lerctr.org> <9eef3e7e964cb33fd163cc2f9300f326@mailbox.ijs.si> <d5bda7f9ad7f86ec35ab6ba5571aa8bc@thebighonker.lerctr.org> <fbf874048b94066aabc4e656e2d3157d@thebighonker.lerctr.org> <c8e77e506bdba2bc1b9dd20ada388879@mailbox.ijs.si> <53D1AB3D.1060205@freebsd.org> <33c478c3120c21d6cb9c325cc1119cf8@thebighonker.lerctr.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 2014-07-24 19:56, Allan Jude wrote:
>>> or better yet:
>>> ssh root@tbh.lerctr.org "zfs send ..." | mbuffer -m 16M | zfs recv
>>> ...
>>> (The misc/mbuffer compensates for bursty zfs reads and writes.
>>> A note to myself: I should suggest to Allan to add mbuffer
>>> in a pipe as used in sysutils/zxfer, instead of patching zxfer
>>> for our local use :)
>>
>> zxfer can already do this, with the -D option
>> I actually use misc/clpbar and get a progress bar as well
>>
>> -D 'bar -s %%size%% -bl 1m -bs 128m'
>>
>> or in your case: -D 'mbuffer -m 16M'
Thank you Allan! It shows it's been a while since the last time
I inspected the guts of zxfer.
2014-07-25 06:43 Larry Rosenman wrote:
> Ok, I did the mbuffer trick, and the SEND side is where the memory
> issue is:
> borg.lerctr.org /home/ler/bin $ tail zfs-send.log
> 23:28:12 15.7G zroot/home@2014-07-24_22:56
> 23:28:13 15.7G zroot/home@2014-07-24_22:56
> 23:28:14 15.7G zroot/home@2014-07-24_22:56
> 23:28:15 15.7G zroot/home@2014-07-24_22:56
> 23:28:16 15.7G zroot/home@2014-07-24_22:56
> 23:28:17 15.7G zroot/home@2014-07-24_22:56
> 23:28:18 15.7G zroot/home@2014-07-24_22:56
> 23:28:19 15.7G zroot/home@2014-07-24_22:56
> 23:28:20 15.8G zroot/home@2014-07-24_22:56
> Write failed: Cannot allocate memory
> borg.lerctr.org /home/ler/bin $
Good information. So the "Write failed: Cannot allocate memory"
on the send side is what is causing the "invalid backup stream"
on the receiving side.
> borg.lerctr.org /home/ler/bin $ tail zfs-recv.log
> cannot receive new filesystem stream: invalid backup stream
> borg.lerctr.org /home/ler/bin $
>
> borg.lerctr.org /home/ler/bin $ cat backup-TBH-ZFS-initial.sh
> #!/bin/sh
> DATE=`date "+%Y-%m-%d_%H:%M"`
> #DATE2=2013-03-24
> #DATE2=`date -v "-1d" "+%Y-%m-%d"`
> # snap the source
> ssh root@tbh.lerctr.org zfs snapshot -r zroot@${DATE}
> # zfs copy the source to here.
> ssh root@tbh.lerctr.org 2>zfs-send.log "zfs send -v -R zroot@${DATE} "
> | \
> mbuffer -m 16M 2>mbuffer.log | \
> zfs recv -F -u -v -d zroot/backups/TBH3 2>zfs-recv.log
> # make sure we NEVER allow the backup stuff to automount.
> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH3| \
> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
> borg.lerctr.org /home/ler/bin $
>
> borg.lerctr.org /home/ler/bin $ ssh tbh vmstat -z
> ITEM SIZE LIMIT USED FREE REQ FAIL
> SLEEP
[...]
> Where to now?
Don't know, I'd guess some network-related memory limit is being hit
on the sending site.
Why not try to decouple the 'zfs send' from a network copy and ssh:
Login to a remote side, do a 'zfs send' to some local temporary file
there, then feed that file to ssh and send it over to a home host,
where it can be piped into some simple program like md5 or 'wc -c'
instead of a zfs recv.
Mark
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?60845088153247fa31c50c9f52ef72cb>