Date: Thu, 24 Jul 2014 23:43:10 -0500 From: Larry Rosenman <ler@lerctr.org> To: Allan Jude <allanjude@freebsd.org>, Mark Martinec <Mark.Martinec+freebsd@ijs.si> Cc: Freebsd fs <freebsd-fs@freebsd.org>, freebsd-current@freebsd.org Subject: Re: zfs send/recv: STILL invalid Backup Stream Message-ID: <33c478c3120c21d6cb9c325cc1119cf8@thebighonker.lerctr.org> In-Reply-To: <53D1AB3D.1060205@freebsd.org> References: <62315eb454a95db636b7764aad3c0f9b@thebighonker.lerctr.org> <53D1448C.40908@freebsd.org> <d0612ef9399b4e95bd09e5271c91505b@thebighonker.lerctr.org> <eefa10e173e2e23a88c85c38221a1c22@thebighonker.lerctr.org> <53D15438.6040105@freebsd.org> <0a9cd451c3b4304d2b9d899fcb3decc3@thebighonker.lerctr.org> <cc635a377e817cb57d27650ed3e558f4@thebighonker.lerctr.org> <3d2aac84d962a703fbf56a864ba5f19c@mailbox.ijs.si> <c93ef94a8e9e076b86a65f4986d0e30d@thebighonker.lerctr.org> <9eef3e7e964cb33fd163cc2f9300f326@mailbox.ijs.si> <d5bda7f9ad7f86ec35ab6ba5571aa8bc@thebighonker.lerctr.org> <fbf874048b94066aabc4e656e2d3157d@thebighonker.lerctr.org> <c8e77e506bdba2bc1b9dd20ada388879@mailbox.ijs.si> <53D1AB3D.1060205@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2014-07-24 19:56, Allan Jude wrote:
> On 2014-07-24 20:46, Mark Martinec wrote:
>> 2014-07-25 01:36 Larry Rosenman wrote:
>>
>>> #!/bin/sh
>>> DATE=`date "+%Y-%m-%d"`
>>> #DATE2=2013-03-24
>>> #DATE2=`date -v "-1d" "+%Y-%m-%d"`
>>> # snap the source
>>> ssh root@tbh.lerctr.org zfs snapshot -r zroot@${DATE}
>>> # zfs copy the source to here.
>>> ssh root@tbh.lerctr.org "zfs send -v -R zroot@${DATE} | \
>>> ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH2\""
>>
>> Btw, this double-ssh looks awkward, why not just:
>>
>> ssh root@tbh.lerctr.org "zfs send ..." | zfs recv ...
>>
>> or better yet:
>>
>> ssh root@tbh.lerctr.org "zfs send ..." | mbuffer -m 16M | zfs recv
>> ...
>>
>> (The misc/mbuffer compensates for bursty zfs reads and writes.
>> A note to myself: I should suggest to Allan to add mbuffer
>> in a pipe as used in sysutils/zxfer, instead of patching zxfer
>> for our local use :)
>>
>> Mark
>> _______________________________________________
>> freebsd-current@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to
>> "freebsd-current-unsubscribe@freebsd.org"
>
> zxfer can already do this, with the -D option
> I actually use misc/clpbar and get a progress bar as well
>
> -D 'bar -s %%size%% -bl 1m -bs 128m'
>
> or in your case: -D 'mbuffer -m 16M'
Ok, I did the mbuffer trick, and the SEND side is where the memory issue
is:
borg.lerctr.org /home/ler/bin $ tail zfs-send.log
23:28:12 15.7G zroot/home@2014-07-24_22:56
23:28:13 15.7G zroot/home@2014-07-24_22:56
23:28:14 15.7G zroot/home@2014-07-24_22:56
23:28:15 15.7G zroot/home@2014-07-24_22:56
23:28:16 15.7G zroot/home@2014-07-24_22:56
23:28:17 15.7G zroot/home@2014-07-24_22:56
23:28:18 15.7G zroot/home@2014-07-24_22:56
23:28:19 15.7G zroot/home@2014-07-24_22:56
23:28:20 15.8G zroot/home@2014-07-24_22:56
Write failed: Cannot allocate memory
borg.lerctr.org /home/ler/bin $
borg.lerctr.org /home/ler/bin $ tail zfs-recv.log
cannot receive new filesystem stream: invalid backup stream
borg.lerctr.org /home/ler/bin $
borg.lerctr.org /home/ler/bin $ cat backup-TBH-ZFS-initial.sh
#!/bin/sh
DATE=`date "+%Y-%m-%d_%H:%M"`
#DATE2=2013-03-24
#DATE2=`date -v "-1d" "+%Y-%m-%d"`
# snap the source
ssh root@tbh.lerctr.org zfs snapshot -r zroot@${DATE}
# zfs copy the source to here.
ssh root@tbh.lerctr.org 2>zfs-send.log "zfs send -v -R zroot@${DATE} "
| \
mbuffer -m 16M 2>mbuffer.log | \
zfs recv -F -u -v -d zroot/backups/TBH3 2>zfs-recv.log
# make sure we NEVER allow the backup stuff to automount.
/sbin/zfs list -H -t filesystem -r zroot/backups/TBH3| \
awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
borg.lerctr.org /home/ler/bin $
borg.lerctr.org /home/ler/bin $ ssh tbh vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 384, 0, 202, 8, 202, 0, 0
UMA Zones: 1664, 0, 202, 0, 202, 0, 0
UMA Slabs: 80, 0, 363320, 44080, 2348572, 0, 0
UMA RCntSlabs: 88, 0, 4484, 16, 4484, 0, 0
UMA Hash: 256, 0, 7, 83, 82, 0, 0
4 Bucket: 32, 0, 1911, 36589, 5255345, 0, 0
6 Bucket: 48, 0, 9406, 3542, 878903, 0, 0
8 Bucket: 64, 0, 42, 3554, 298443, 11, 0
12 Bucket: 96, 0, 93, 2572, 166067, 0, 0
16 Bucket: 128, 0, 30447, 987, 301403, 0, 0
32 Bucket: 256, 0, 352, 4658, 1157489, 50, 0
64 Bucket: 512, 0, 13669, 995, 1113780,268080,
0
128 Bucket: 1024, 0, 3646, 822, 524977, 0, 0
256 Bucket: 2048, 0, 3648, 114, 482627,59834,
0
vmem btag: 56, 0, 208448, 49779, 758362,1821, 0
VM OBJECT: 256, 0, 98960, 1570, 4440323, 0, 0
RADIX NODE: 144, 0, 235166, 29650,22669417, 0, 0
MAP: 240, 0, 3, 61, 3, 0, 0
KMAP ENTRY: 128, 0, 10, 269, 10, 0, 0
MAP ENTRY: 128, 0, 11828, 24442,12463199, 0, 0
VMSPACE: 448, 0, 103, 311, 96786, 0, 0
fakepg: 104, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 368, 0, 368, 0, 0
16: 16, 0, 264961, 9382,56032463, 0, 0
32: 32, 0, 155626, 2874,55177821, 0, 0
64: 64, 0, 123597, 672111,53838666, 0, 0
128: 128, 0, 159107, 58792,82084329, 0, 0
256: 256, 0, 97004, 223276,48661927, 0, 0
512: 512, 0, 737, 3991,33323191, 0, 0
1024: 1024, 0, 1367, 1389, 2330023, 0, 0
2048: 2048, 0, 271, 449,16353342, 0, 0
4096: 4096, 0, 28044, 27448, 997566, 0, 0
SLEEPQUEUE: 80, 0, 613, 658, 613, 0, 0
64 pcpu: 8, 0, 1884, 1188, 1884, 0, 0
Files: 80, 0, 861, 1639, 5410116, 0, 0
rl_entry: 40, 0, 211, 1889, 211, 0, 0
TURNSTILE: 136, 0, 613, 307, 613, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1208, 0, 138, 114, 97778, 0, 0
THREAD: 1168, 0, 517, 95, 2429, 0, 0
cpuset: 72, 0, 270, 280, 426, 0, 0
cyclic_id_cache: 64, 0, 0, 0, 0, 0, 0
audit_record: 1248, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 6519810, 4080, 3257, 9746764, 0,
0
mbuf: 256, 6519810, 1025, 3278,259201769, 0,
0
mbuf_cluster: 2048, 1018718, 7337, 13, 7337, 0,
0
mbuf_jumbo_page: 4096, 509359, 0, 809,45525783, 0, 0
mbuf_jumbo_9k: 9216, 150921, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 84893, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 2510, 116647, 0, 0
dtrace_state_cache: 4096, 0, 0, 0, 0, 0, 0
g_bio: 248, 0, 0, 1680,48127406, 0, 0
ttyinq: 160, 0, 120, 630, 2460, 0, 0
DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0
ttyoutq: 256, 0, 64, 671, 1279, 0, 0
ata_request: 336, 0, 0, 242, 42892, 0, 0
vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0
cryptop: 88, 0, 0, 0, 0, 0, 0
cryptodesc: 72, 0, 0, 0, 0, 0, 0
FPU_save_area: 512, 0, 0, 0, 0, 0, 0
taskq_zone: 48, 0, 0, 3071, 250215, 0, 0
taskq_zone: 48, 0, 0, 0, 0, 0, 0
VNODE: 472, 0, 113296, 62392, 2709794, 0, 0
VNODEPOLL: 112, 0, 355, 660, 357, 0, 0
BUF TRIE: 144, 0, 0, 105354, 0, 0, 0
S VFS Cache: 108, 0, 118644, 56881, 2428046, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 1110, 144894, 804385, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 172,30294048, 0, 0
DIRHASH: 1024, 0, 0, 0, 0, 0, 0
NCLNODE: 528, 0, 0, 0, 0, 0, 0
Mountpoints: 816, 0, 11, 19, 11, 0, 0
range_seg_cache: 64, 0, 59688, 6094, 4438296, 0, 0
zio_cache: 920, 0, 1, 42591,138793685, 0,
0
zio_link_cache: 48, 0, 0, 44073,97697152, 0, 0
zio_buf_512: 512, 0, 411934, 265850, 1785999, 0, 0
zio_data_buf_512: 512, 0, 154726, 20594, 797483, 0, 0
zio_buf_1024: 1024, 0, 3061, 175, 74166, 0, 0
zio_data_buf_1024: 1024, 0, 69831, 9029, 333451, 0, 0
zio_buf_1536: 1536, 0, 1372, 96, 93985, 0, 0
zio_data_buf_1536: 1536, 0, 29821, 3103, 190819, 0, 0
zio_buf_2048: 2048, 0, 523, 47, 281620, 0, 0
zio_data_buf_2048: 2048, 0, 17707, 2389, 128398, 0, 0
zio_buf_2560: 2560, 0, 390, 28, 22269, 0, 0
zio_data_buf_2560: 2560, 0, 12713, 28, 108546, 0, 0
zio_buf_3072: 3072, 0, 182, 22, 7255, 0, 0
zio_data_buf_3072: 3072, 0, 8798, 34, 79372, 0, 0
zio_buf_3584: 3584, 0, 206, 11, 50516, 0, 0
zio_data_buf_3584: 3584, 0, 6275, 11, 64145, 0, 0
zio_buf_4096: 4096, 0, 1552, 416,18817937, 0, 0
zio_data_buf_4096: 4096, 0, 4625, 6, 59910, 0, 0
zio_buf_5120: 5120, 0, 133, 9, 8048, 0, 0
zio_data_buf_5120: 5120, 0, 6326, 11, 77387, 0, 0
zio_buf_6144: 6144, 0, 86, 11, 4503, 0, 0
zio_data_buf_6144: 6144, 0, 4632, 13, 58064, 0, 0
zio_buf_7168: 7168, 0, 61, 13, 48012, 0, 0
zio_data_buf_7168: 7168, 0, 3090, 13, 44010, 0, 0
zio_buf_8192: 8192, 0, 24, 29, 1857848, 0, 0
zio_data_buf_8192: 8192, 0, 2418, 11, 42210, 0, 0
zio_buf_10240: 10240, 0, 49, 7, 3311, 0, 0
zio_data_buf_10240: 10240, 0, 3611, 15, 59310, 0, 0
zio_buf_12288: 12288, 0, 44, 14, 999494, 0, 0
zio_data_buf_12288: 12288, 0, 2340, 13, 53847, 0, 0
zio_buf_14336: 14336, 0, 23, 10, 4184, 0, 0
zio_data_buf_14336: 14336, 0, 1852, 15, 36058, 0, 0
zio_buf_16384: 16384, 0, 72595, 164, 3891311, 0, 0
zio_data_buf_16384: 16384, 0, 1707, 11, 33236, 0, 0
zio_buf_20480: 20480, 0, 201, 261, 832993, 0, 0
zio_data_buf_20480: 20480, 0, 1932, 12, 43873, 0, 0
zio_buf_24576: 24576, 0, 16, 11, 365481, 0, 0
zio_data_buf_24576: 24576, 0, 1852, 13, 31773, 0, 0
zio_buf_28672: 28672, 0, 9, 25, 731759, 0, 0
zio_data_buf_28672: 28672, 0, 1169, 10, 27692, 0, 0
zio_buf_32768: 32768, 0, 9, 14, 112680, 0, 0
zio_data_buf_32768: 32768, 0, 962, 13, 20775, 0, 0
zio_buf_36864: 36864, 0, 5, 14, 112387, 0, 0
zio_data_buf_36864: 36864, 0, 701, 13, 15139, 0, 0
zio_buf_40960: 40960, 0, 4, 11, 115602, 0, 0
zio_data_buf_40960: 40960, 0, 720, 10, 12360, 0, 0
zio_buf_45056: 45056, 0, 48, 12, 102919, 0, 0
zio_data_buf_45056: 45056, 0, 434, 14, 10533, 0, 0
zio_buf_49152: 49152, 0, 188, 12, 113099, 0, 0
zio_data_buf_49152: 49152, 0, 373, 10, 11797, 0, 0
zio_buf_53248: 53248, 0, 24, 13, 157215, 0, 0
zio_data_buf_53248: 53248, 0, 347, 15, 7265, 0, 0
zio_buf_57344: 57344, 0, 1, 11, 154125, 0, 0
zio_data_buf_57344: 57344, 0, 274, 14, 7110, 0, 0
zio_buf_61440: 61440, 0, 2, 10, 163599, 0, 0
zio_data_buf_61440: 61440, 0, 180, 11, 5565, 0, 0
zio_buf_65536: 65536, 0, 1, 11, 186705, 0, 0
zio_data_buf_65536: 65536, 0, 213, 11, 7273, 0, 0
zio_buf_69632: 69632, 0, 2, 14, 145850, 0, 0
zio_data_buf_69632: 69632, 0, 176, 8, 5232, 0, 0
zio_buf_73728: 73728, 0, 1, 13, 136031, 0, 0
zio_data_buf_73728: 73728, 0, 127, 12, 4816, 0, 0
zio_buf_77824: 77824, 0, 1, 22, 162862, 0, 0
zio_data_buf_77824: 77824, 0, 136, 14, 4207, 0, 0
zio_buf_81920: 81920, 0, 0, 149, 195810, 0, 0
zio_data_buf_81920: 81920, 0, 99, 15, 4936, 0, 0
zio_buf_86016: 86016, 0, 1, 216, 150971, 0, 0
zio_data_buf_86016: 86016, 0, 103, 14, 33989, 0, 0
zio_buf_90112: 90112, 0, 1, 120, 75823, 0, 0
zio_data_buf_90112: 90112, 0, 145, 11, 3517, 0, 0
zio_buf_94208: 94208, 0, 0, 20, 72296, 0, 0
zio_data_buf_94208: 94208, 0, 149, 11, 3315, 0, 0
zio_buf_98304: 98304, 0, 0, 19, 68257, 0, 0
zio_data_buf_98304: 98304, 0, 79, 11, 4092, 0, 0
zio_buf_102400: 102400, 0, 1, 19, 62463, 0, 0
zio_data_buf_102400: 102400, 0, 92, 12, 2940, 0, 0
zio_buf_106496: 106496, 0, 0, 15, 203444, 0, 0
zio_data_buf_106496: 106496, 0, 62, 14, 2869, 0, 0
zio_buf_110592: 110592, 0, 0, 10, 184072, 0, 0
zio_data_buf_110592: 110592, 0, 152, 12, 2463, 0, 0
zio_buf_114688: 114688, 0, 1, 12, 65538, 0, 0
zio_data_buf_114688: 114688, 0, 115, 15, 3609, 0, 0
zio_buf_118784: 118784, 0, 0, 13, 73524, 0, 0
zio_data_buf_118784: 118784, 0, 57, 12, 2092, 0, 0
zio_buf_122880: 122880, 0, 0, 23, 101816, 0, 0
zio_data_buf_122880: 122880, 0, 46, 15, 2131, 0, 0
zio_buf_126976: 126976, 0, 0, 13, 99774, 0, 0
zio_data_buf_126976: 126976, 0, 46, 15, 1927, 0, 0
zio_buf_131072: 131072, 0, 21, 97, 715685, 0, 0
zio_data_buf_131072: 131072, 0, 64623, 8, 1133068, 0, 0
lz4_ctx: 16384, 0, 0, 38, 2874956, 0, 0
sa_cache: 80, 0, 113220, 62680, 2719738, 0, 0
dnode_t: 744, 0, 369347, 391598, 903612, 0, 0
dmu_buf_impl_t: 224, 0, 459185, 519692, 3494852, 0, 0
arc_buf_hdr_t: 216, 0, 697777, 122573, 4738762, 0, 0
arc_buf_t: 72, 0, 541440, 121695, 5738782, 0, 0
zil_lwb_cache: 192, 0, 7, 5073, 90526, 0, 0
zfs_znode_cache: 368, 0, 113220, 62340, 2709584, 0, 0
pipe: 744, 0, 113, 177, 84427, 0, 0
procdesc: 128, 0, 0, 0, 0, 0, 0
ksiginfo: 112, 0, 1133, 2507,10649838, 0, 0
itimer: 352, 0, 1, 32, 1, 0, 0
KNOTE: 128, 0, 379, 1388,26313314, 0, 0
socket: 696, 523340, 243, 317, 827216, 0, 0
unpcb: 240, 523344, 152, 648, 171731, 0, 0
ipq: 56, 31879, 0, 2059, 385, 0, 0
udp_inpcb: 392, 523340, 19, 421, 567701, 0, 0
udpcb: 16, 523586, 19, 2993, 567701, 0, 0
tcp_inpcb: 392, 523340, 73, 457, 87773, 0, 0
tcpcb: 1024, 523340, 67, 245, 87773, 0, 0
tcptw: 88, 27810, 6, 1254, 29354, 0, 0
syncache: 160, 15375, 0, 800, 57377, 0, 0
hostcache: 136, 15370, 134, 620, 4951, 0, 0
tcpreass: 40, 63700, 0, 3100, 13421, 0, 0
sackhole: 32, 0, 0, 2125, 18903, 0, 0
sctp_ep: 1408, 523340, 0, 0, 0, 0, 0
sctp_asoc: 2352, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80012, 0, 1328, 8, 0, 0
sctp_raddr: 728, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400026, 0, 0, 0, 0, 0
sctp_readq: 104, 400026, 0, 0, 0, 0, 0
sctp_stream_msg_out: 104, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400000, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0
udplite_inpcb: 392, 523340, 0, 0, 0, 0, 0
ripcb: 392, 523340, 2, 58, 3, 0, 0
rtentry: 200, 0, 32, 408, 34, 0, 0
selfd: 56, 0, 356, 2200,137062686, 0,
0
SWAPMETA: 288, 2037438, 0, 0, 0, 0,
0
borg.lerctr.org /home/ler/bin $
last pid: 97813; load averages: 7.79, 8.41, 8.24 up 1+11:45:45
23:42:11
118 processes: 7 running, 109 sleeping, 2 stopped
Mem: 982M Active, 1172M Inact, 13G Wired, 148M Cache, 637M Free
ARC: 11G Total, 4394M MFU, 5813M MRU, 1029K Anon, 181M Header, 469M
Other
Swap: 18G Total, 18G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
97782 boinc 1 155 i31 91992K 26008K CPU1 1 0:40 100.00%
wcgrid_f
96977 boinc 1 155 i31 375M 165M STOP 4 11:30 61.87%
wcgrid_c
89981 boinc 1 155 i31 99308K 63248K CPU5 5 91:25 51.76%
wcgrid_f
92270 boinc 1 155 i31 98M 64556K CPU3 3 62:55 50.88%
wcgrid_f
92269 boinc 1 155 i31 99372K 63324K CPU4 4 63:49 50.20%
wcgrid_f
92271 boinc 1 155 i31 99340K 63288K CPU0 0 64:00 48.19%
wcgrid_f
89772 boinc 1 155 i31 99888K 63832K CPU7 7 92:52 41.36%
wcgrid_f
89781 boinc 1 155 i31 99732K 63588K nanslp 5 94:54 38.18%
wcgrid_f
85619 root 1 52 0 181M 97232K select 3 2:56 21.19%
perl5
97806 root 1 28 0 44600K 8028K kqread 0 0:00 0.78%
indexer-
837 clamav 3 20 0 486M 399M uwait 5 12:42 0.20%
clamd
70563 boinc 2 155 i31 101M 51436K nanslp 7 202:02 0.00%
setiatho
68437 boinc 2 155 i31 105M 52516K nanslp 3 185:45 0.00%
setiatho
83114 boinc 2 155 i31 105M 51880K nanslp 4 103:15 0.00%
setiatho
85300 boinc 2 155 i31 101M 51460K nanslp 4 78:55 0.00%
setiatho
86148 boinc 2 155 i31 105M 51492K nanslp 3 71:54 0.00%
setiatho
87150 boinc 2 155 i31 95112K 46940K nanslp 4 30:37 0.00%
setiatho
87451 boinc 2 155 i31 95112K 46224K nanslp 3 28:07 0.00%
setiatho
borg.lerctr.org /home/ler/bin $
Where to now?
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?33c478c3120c21d6cb9c325cc1119cf8>