Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 24 Jul 2014 15:11:31 -0500
From:      Larry Rosenman <ler@lerctr.org>
To:        Allan Jude <allanjude@freebsd.org>
Cc:        Freebsd fs <freebsd-fs@freebsd.org>, freebsd-current@freebsd.org
Subject:   Re: zfs send/recv: STILL invalid Backup Stream
Message-ID:  <6e382c7d7efc8b4ddc3b770985eac1e0@thebighonker.lerctr.org>
In-Reply-To: <53D1678C.4000007@freebsd.org>
References:  <62315eb454a95db636b7764aad3c0f9b@thebighonker.lerctr.org> <53D1448C.40908@freebsd.org> <d0612ef9399b4e95bd09e5271c91505b@thebighonker.lerctr.org> <eefa10e173e2e23a88c85c38221a1c22@thebighonker.lerctr.org> <53D15438.6040105@freebsd.org> <0a9cd451c3b4304d2b9d899fcb3decc3@thebighonker.lerctr.org> <cc635a377e817cb57d27650ed3e558f4@thebighonker.lerctr.org> <3d2aac84d962a703fbf56a864ba5f19c@mailbox.ijs.si> <c93ef94a8e9e076b86a65f4986d0e30d@thebighonker.lerctr.org> <53D1678C.4000007@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On 2014-07-24 15:07, Allan Jude wrote:
> On 2014-07-24 15:57, Larry Rosenman wrote:
>> On 2014-07-24 14:53, Mark Martinec wrote:
>>> 2014-07-24 21:31, Larry Rosenman wrote:
>>>> borg.lerctr.org /home/ler # zxfer -dFkPvs -g 376 -O
>>>> root@tbh.lerctr.org -R zroot  zroot/backups/TBH
>>>> Creating recursive snapshot zroot@zxfer_26699_20140724135840.
>>>> Checking grandfather status of all snapshots marked for deletion...
>>>> Grandfather check passed.
>>>> Sending zroot@zxfer_26699_20140724135840 to zroot/backups/TBH/zroot.
>>>> Sending zroot/ROOT@zxfer_26699_20140724135840 to
>>>> zroot/backups/TBH/zroot/ROOT.
>>>> Sending zroot/ROOT/default@zxfer_23699_20140724134435 to
>>>> zroot/backups/TBH/zroot/ROOT/default.
>>>> Sending zroot/ROOT/default@zxfer_26699_20140724135840 to
>>>> zroot/backups/TBH/zroot/ROOT/default.
>>>>   (incremental to zroot/ROOT/default@zxfer_23699_20140724134435.)
>>>> Sending zroot/home@zxfer_26699_20140724135840 to
>>>> zroot/backups/TBH/zroot/home.
>>> 
>>>> Write failed: Cannot allocate memory
>>>   ====================================
>>> 
>>>> cannot receive new filesystem stream: invalid backup stream
>>>> Error when zfs send/receiving.
>>>> borg.lerctr.org /home/ler #
>>>> 
>>>> well that's different.......
>>> 
>>> Sounds familiar, check my posting of today and links therein:
>>> 
>>>   
>>> http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html
>>> 
>>> Mark
>> I'm not using netgraph to the best of my knowledge....
>> and the only fails on the SENDING host are:
>> 8 Bucket:                64,      0,      41,    3555,  257774,  11,   
>> 0
>> 12 Bucket:               96,      0,      96,    2569,  123653,   0,   
>> 0
>> 16 Bucket:              128,      0,   17195,     506,  215573,   0,   
>> 0
>> 32 Bucket:              256,      0,     340,    4670,  900638,  50,   
>> 0
>> 64 Bucket:              512,      0,   10691,     365,  546888,185232, 
>>   0
>> 128 Bucket:            1024,      0,    3563,     905,  348419,   0,   
>> 0
>> 256 Bucket:            2048,      0,    2872,     162,  249995,59834,  
>>  0
>> vmem btag:               56,      0,  192811,   51500,  502264,1723,   
>> 0
>> 
>> 
> 
> I regularly use zxfer to transfer 500+ GiB datasets over the internet.
> This week I actually replicated a 2.1 TiB dataset with zxfer without 
> issue.
> 
> I wonder which thing is running out of memory. Is there a delay while 
> it
> is 'running out of memory', or does it fail immediately? Does running
> top while it is working on running out of memory reveal anything?
> 
> I would expect to use up a lot of memory while doing deduplication, but
> not otherwise.
> 
> Note: I most often use openssh-portable rather than base ssh for
> replication, as I enable the nonecipher to reduce CPU usage, and adjust
> the TcpRcvBuf upwards to actually saturate a gigabit over the internet.

I wasn't watching exactly what it was doing, but the sending box has 16G 
and 18G Swap and swap
has NOT been touched.

last pid: 74288;  load averages:  4.70,  5.61,  5.91    up 1+03:14:18  
15:10:44
115 processes: 3 running, 112 sleeping
CPU:  0.6% user, 33.3% nice,  0.6% system,  0.1% interrupt, 65.4% idle
Mem: 847M Active, 761M Inact, 14G Wired, 4616K Cache, 357M Free
ARC: 12G Total, 6028M MFU, 5281M MRU, 3152K Anon, 120M Header, 688M 
Other
Swap: 18G Total, 18G Free

so I have zero idea where to go here.


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c)     E-Mail: ler@lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6e382c7d7efc8b4ddc3b770985eac1e0>