From owner-freebsd-fs@FreeBSD.ORG Thu Jul 24 20:11:44 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DD0EB650; Thu, 24 Jul 2014 20:11:44 +0000 (UTC) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A82282376; Thu, 24 Jul 2014 20:11:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=3UiDi1+aocOnaMg1FHQjMIAViSgjux9q+dCNC03jEkg=; b=Qnh4MutrKnlLnOjNmdIc5Yi2bRiUvBTAjG3VOwFXHZrC/7f5lKdCz0tj5E32IEsDucuvCrkusC0rIohhWIR8tlWcacBEncyA7Sv1RyMRpGDqEfFdjP5ujyHTqT9S94Ti5m46+J9zn+yVCRwAWRllxZvMe3ckRBvMZRmSHgPQ5LE=; Received: from localhost.lerctr.org ([127.0.0.1]:30108 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.83 (FreeBSD)) (envelope-from ) id 1XAPMR-000JLE-MY; Thu, 24 Jul 2014 15:11:43 -0500 Received: from host.alcatel.com ([198.205.55.139]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Thu, 24 Jul 2014 15:11:31 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 24 Jul 2014 15:11:31 -0500 From: Larry Rosenman To: Allan Jude Subject: Re: zfs send/recv: STILL invalid Backup Stream In-Reply-To: <53D1678C.4000007@freebsd.org> References: <62315eb454a95db636b7764aad3c0f9b@thebighonker.lerctr.org> <53D1448C.40908@freebsd.org> <53D15438.6040105@freebsd.org> <0a9cd451c3b4304d2b9d899fcb3decc3@thebighonker.lerctr.org> <3d2aac84d962a703fbf56a864ba5f19c@mailbox.ijs.si> <53D1678C.4000007@freebsd.org> Message-ID: <6e382c7d7efc8b4ddc3b770985eac1e0@thebighonker.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.0.1 X-Spam-Score: -2.9 (--) X-LERCTR-Spam-Score: -2.9 (--) X-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.001 X-LERCTR-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.001 Cc: Freebsd fs , freebsd-current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 20:11:45 -0000 On 2014-07-24 15:07, Allan Jude wrote: > On 2014-07-24 15:57, Larry Rosenman wrote: >> On 2014-07-24 14:53, Mark Martinec wrote: >>> 2014-07-24 21:31, Larry Rosenman wrote: >>>> borg.lerctr.org /home/ler # zxfer -dFkPvs -g 376 -O >>>> root@tbh.lerctr.org -R zroot zroot/backups/TBH >>>> Creating recursive snapshot zroot@zxfer_26699_20140724135840. >>>> Checking grandfather status of all snapshots marked for deletion... >>>> Grandfather check passed. >>>> Sending zroot@zxfer_26699_20140724135840 to zroot/backups/TBH/zroot. >>>> Sending zroot/ROOT@zxfer_26699_20140724135840 to >>>> zroot/backups/TBH/zroot/ROOT. >>>> Sending zroot/ROOT/default@zxfer_23699_20140724134435 to >>>> zroot/backups/TBH/zroot/ROOT/default. >>>> Sending zroot/ROOT/default@zxfer_26699_20140724135840 to >>>> zroot/backups/TBH/zroot/ROOT/default. >>>> (incremental to zroot/ROOT/default@zxfer_23699_20140724134435.) >>>> Sending zroot/home@zxfer_26699_20140724135840 to >>>> zroot/backups/TBH/zroot/home. >>> >>>> Write failed: Cannot allocate memory >>> ==================================== >>> >>>> cannot receive new filesystem stream: invalid backup stream >>>> Error when zfs send/receiving. >>>> borg.lerctr.org /home/ler # >>>> >>>> well that's different....... >>> >>> Sounds familiar, check my posting of today and links therein: >>> >>> >>> http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html >>> >>> Mark >> I'm not using netgraph to the best of my knowledge.... >> and the only fails on the SENDING host are: >> 8 Bucket: 64, 0, 41, 3555, 257774, 11, >> 0 >> 12 Bucket: 96, 0, 96, 2569, 123653, 0, >> 0 >> 16 Bucket: 128, 0, 17195, 506, 215573, 0, >> 0 >> 32 Bucket: 256, 0, 340, 4670, 900638, 50, >> 0 >> 64 Bucket: 512, 0, 10691, 365, 546888,185232, >> 0 >> 128 Bucket: 1024, 0, 3563, 905, 348419, 0, >> 0 >> 256 Bucket: 2048, 0, 2872, 162, 249995,59834, >> 0 >> vmem btag: 56, 0, 192811, 51500, 502264,1723, >> 0 >> >> > > I regularly use zxfer to transfer 500+ GiB datasets over the internet. > This week I actually replicated a 2.1 TiB dataset with zxfer without > issue. > > I wonder which thing is running out of memory. Is there a delay while > it > is 'running out of memory', or does it fail immediately? Does running > top while it is working on running out of memory reveal anything? > > I would expect to use up a lot of memory while doing deduplication, but > not otherwise. > > Note: I most often use openssh-portable rather than base ssh for > replication, as I enable the nonecipher to reduce CPU usage, and adjust > the TcpRcvBuf upwards to actually saturate a gigabit over the internet. I wasn't watching exactly what it was doing, but the sending box has 16G and 18G Swap and swap has NOT been touched. last pid: 74288; load averages: 4.70, 5.61, 5.91 up 1+03:14:18 15:10:44 115 processes: 3 running, 112 sleeping CPU: 0.6% user, 33.3% nice, 0.6% system, 0.1% interrupt, 65.4% idle Mem: 847M Active, 761M Inact, 14G Wired, 4616K Cache, 357M Free ARC: 12G Total, 6028M MFU, 5281M MRU, 3152K Anon, 120M Header, 688M Other Swap: 18G Total, 18G Free so I have zero idea where to go here. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688