Date: Thu, 24 Jul 2014 15:44:29 -0500 From: Larry Rosenman <ler@lerctr.org> To: Mark Martinec <Mark.Martinec+freebsd@ijs.si> Cc: Freebsd fs <freebsd-fs@freebsd.org>, freebsd-current@freebsd.org Subject: Re: zfs send/recv: STILL invalid Backup Stream Message-ID: <d5bda7f9ad7f86ec35ab6ba5571aa8bc@thebighonker.lerctr.org> In-Reply-To: <9eef3e7e964cb33fd163cc2f9300f326@mailbox.ijs.si> References: <62315eb454a95db636b7764aad3c0f9b@thebighonker.lerctr.org> <53D1448C.40908@freebsd.org> <d0612ef9399b4e95bd09e5271c91505b@thebighonker.lerctr.org> <eefa10e173e2e23a88c85c38221a1c22@thebighonker.lerctr.org> <53D15438.6040105@freebsd.org> <0a9cd451c3b4304d2b9d899fcb3decc3@thebighonker.lerctr.org> <cc635a377e817cb57d27650ed3e558f4@thebighonker.lerctr.org> <3d2aac84d962a703fbf56a864ba5f19c@mailbox.ijs.si> <c93ef94a8e9e076b86a65f4986d0e30d@thebighonker.lerctr.org> <9eef3e7e964cb33fd163cc2f9300f326@mailbox.ijs.si>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2014-07-24 15:26, Mark Martinec wrote: > 2014-07-24 21:57, Larry Rosenman wrote: > >>>>> Sending zroot/home@zxfer_26699_20140724135840 to >>>>> zroot/backups/TBH/zroot/home. >>>> Write failed: Cannot allocate memory >>> ==================================== >>>> cannot receive new filesystem stream: invalid backup stream >>>> Error when zfs send/receiving. >>>> borg.lerctr.org /home/ler # >>>> >>>> well that's different....... >>> >>> Sounds familiar, check my posting of today and links therein: >>> >>> http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html > >> I'm not using netgraph to the best of my knowledge.... > > Check to make sure: > vmstat -z | fgrep NetGraph > >> and the only fails on the SENDING host are: >> 8 Bucket: 64, 0, 41, 3555, 257774, 11, >> 0 >> 12 Bucket: 96, 0, 96, 2569, 123653, 0, >> 0 >> 16 Bucket: 128, 0, 17195, 506, 215573, 0, >> 0 >> 32 Bucket: 256, 0, 340, 4670, 900638, 50, >> 0 >> 64 Bucket: 512, 0, 10691, 365, 546888,185232, >> 0 >> 128 Bucket: 1024, 0, 3563, 905, 348419, 0, >> 0 >> 256 Bucket: 2048, 0, 2872, 162, 249995,59834, >> 0 >> vmem btag: 56, 0, 192811, 51500, 502264,1723, >> 0 > > > Adam Vande More gave other suggestions on that thread from 2011: > > > http://lists.freebsd.org/pipermail/freebsd-emulation/2011-July/008971.html > > > Mark thebighonker.lerctr.org /home/ler $ vmstat -z|fgrep NetGraph thebighonker.lerctr.org /home/ler $ This on physical hardware, FWIW. thebighonker.lerctr.org /home/ler $ vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 202, 8, 202, 0, 0 UMA Zones: 1664, 0, 202, 0, 202, 0, 0 UMA Slabs: 80, 0, 361152, 38448, 1561588, 0, 0 UMA RCntSlabs: 88, 0, 4360, 5, 4360, 0, 0 UMA Hash: 256, 0, 8, 82, 82, 0, 0 4 Bucket: 32, 0, 4816, 33684, 2974043, 0, 0 6 Bucket: 48, 0, 6118, 2514, 624434, 0, 0 8 Bucket: 64, 0, 161, 3435, 265443, 11, 0 12 Bucket: 96, 0, 177, 2488, 130226, 0, 0 16 Bucket: 128, 0, 20301, 531, 224788, 0, 0 32 Bucket: 256, 0, 433, 4577, 945047, 50, 0 64 Bucket: 512, 0, 11137, 575, 579209,211292, 0 128 Bucket: 1024, 0, 3509, 959, 369839, 0, 0 256 Bucket: 2048, 0, 2847, 187, 269044,59834, 0 vmem btag: 56, 0, 209393, 34918, 557576,1723, 0 VM OBJECT: 256, 0, 91384, 791, 3480613, 0, 0 RADIX NODE: 144, 0, 227133, 37683,16029821, 0, 0 MAP: 240, 0, 3, 61, 3, 0, 0 KMAP ENTRY: 128, 0, 10, 269, 10, 0, 0 MAP ENTRY: 128, 0, 12666, 23604, 9357932, 0, 0 VMSPACE: 448, 0, 108, 306, 75357, 0, 0 fakepg: 104, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 368, 0, 368, 0, 0 16: 16, 0, 267839, 2488,45160196, 0, 0 32: 32, 0, 148139, 1486,39174301, 0, 0 64: 64, 0, 291630, 504078,40379996, 0, 0 128: 128, 0, 182409, 35490,63455920, 0, 0 256: 256, 0, 89368, 230912,35841036, 0, 0 512: 512, 0, 741, 3987,24215854, 0, 0 1024: 1024, 0, 1371, 1385, 1726440, 0, 0 2048: 2048, 0, 277, 417,10240207, 0, 0 4096: 4096, 0, 38306, 17186, 913638, 0, 0 SLEEPQUEUE: 80, 0, 613, 658, 613, 0, 0 64 pcpu: 8, 0, 1884, 1188, 1884, 0, 0 Files: 80, 0, 829, 1571, 4279893, 0, 0 rl_entry: 40, 0, 209, 1891, 209, 0, 0 TURNSTILE: 136, 0, 613, 307, 613, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1208, 0, 131, 121, 76219, 0, 0 THREAD: 1168, 0, 519, 93, 2039, 0, 0 cpuset: 72, 0, 270, 280, 426, 0, 0 cyclic_id_cache: 64, 0, 0, 0, 0, 0, 0 audit_record: 1248, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 6519810, 4081, 3256, 4910227, 0, 0 mbuf: 256, 6519810, 1052, 3251,93444045, 0, 0 mbuf_cluster: 2048, 1018718, 7337, 13, 7337, 0, 0 mbuf_jumbo_page: 4096, 509359, 12, 673,15015378, 0, 0 mbuf_jumbo_9k: 9216, 150921, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 84893, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 2510, 85094, 0, 0 dtrace_state_cache: 4096, 0, 0, 0, 0, 0, 0 g_bio: 248, 0, 0, 1600,29986631, 0, 0 ttyinq: 160, 0, 180, 570, 2100, 0, 0 DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0 ttyoutq: 256, 0, 95, 640, 1093, 0, 0 ata_request: 336, 0, 0, 242, 33344, 0, 0 vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 cryptop: 88, 0, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0, 0 FPU_save_area: 512, 0, 0, 0, 0, 0, 0 taskq_zone: 48, 0, 1, 3070, 194184, 0, 0 taskq_zone: 48, 0, 0, 0, 0, 0, 0 VNODE: 472, 0, 109527, 66161, 2663226, 0, 0 VNODEPOLL: 112, 0, 8, 762, 10, 0, 0 BUF TRIE: 144, 0, 0, 105354, 0, 0, 0 S VFS Cache: 108, 0, 114973, 60552, 2293717, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 468, 145536, 800279, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 156,23968745, 0, 0 DIRHASH: 1024, 0, 0, 0, 0, 0, 0 NCLNODE: 528, 0, 0, 0, 0, 0, 0 Mountpoints: 816, 0, 11, 19, 11, 0, 0 range_seg_cache: 64, 0, 58377, 7157, 3500156, 0, 0 zio_cache: 920, 0, 1, 42591,104595517, 0, 0 zio_link_cache: 48, 0, 0, 44073,72482492, 0, 0 zio_buf_512: 512, 0, 552060, 193836, 1557459, 0, 0 zio_data_buf_512: 512, 0, 114904, 2056, 526943, 0, 0 zio_buf_1024: 1024, 0, 4246, 58, 56140, 0, 0 zio_data_buf_1024: 1024, 0, 51117, 1255, 211875, 0, 0 zio_buf_1536: 1536, 0, 1957, 59, 72179, 0, 0 zio_data_buf_1536: 1536, 0, 22199, 719, 127817, 0, 0 zio_buf_2048: 2048, 0, 691, 43, 217643, 0, 0 zio_data_buf_2048: 2048, 0, 13419, 1147, 87149, 0, 0 zio_buf_2560: 2560, 0, 455, 20, 16265, 0, 0 zio_data_buf_2560: 2560, 0, 9335, 493, 73967, 0, 0 zio_buf_3072: 3072, 0, 245, 16, 4658, 0, 0 zio_data_buf_3072: 3072, 0, 6940, 512, 53932, 0, 0 zio_buf_3584: 3584, 0, 248, 10, 38558, 0, 0 zio_data_buf_3584: 3584, 0, 5076, 352, 43162, 0, 0 zio_buf_4096: 4096, 0, 464, 729,14217630, 0, 0 zio_data_buf_4096: 4096, 0, 3857, 288, 41192, 0, 0 zio_buf_5120: 5120, 0, 167, 9, 6442, 0, 0 zio_data_buf_5120: 5120, 0, 6025, 461, 51297, 0, 0 zio_buf_6144: 6144, 0, 115, 6, 3196, 0, 0 zio_data_buf_6144: 6144, 0, 4453, 309, 38753, 0, 0 zio_buf_7168: 7168, 0, 85, 12, 36945, 0, 0 zio_data_buf_7168: 7168, 0, 3373, 273, 29007, 0, 0 zio_buf_8192: 8192, 0, 44, 39, 1373975, 0, 0 zio_data_buf_8192: 8192, 0, 2543, 205, 28736, 0, 0 zio_buf_10240: 10240, 0, 66, 7, 2534, 0, 0 zio_data_buf_10240: 10240, 0, 3986, 275, 39568, 0, 0 zio_buf_12288: 12288, 0, 71, 15, 654156, 0, 0 zio_data_buf_12288: 12288, 0, 2937, 209, 37077, 0, 0 zio_buf_14336: 14336, 0, 32, 9, 3164, 0, 0 zio_data_buf_14336: 14336, 0, 2236, 194, 23959, 0, 0 zio_buf_16384: 16384, 0, 72627, 226, 2784131, 0, 0 zio_data_buf_16384: 16384, 0, 1778, 155, 22372, 0, 0 zio_buf_20480: 20480, 0, 210, 14, 598092, 0, 0 zio_data_buf_20480: 20480, 0, 2724, 216, 29228, 0, 0 zio_buf_24576: 24576, 0, 18, 14, 270172, 0, 0 zio_data_buf_24576: 24576, 0, 2101, 173, 21241, 0, 0 zio_buf_28672: 28672, 0, 10, 54, 503029, 0, 0 zio_data_buf_28672: 28672, 0, 1406, 113, 18667, 0, 0 zio_buf_32768: 32768, 0, 10, 11, 77998, 0, 0 zio_data_buf_32768: 32768, 0, 1088, 96, 14116, 0, 0 zio_buf_36864: 36864, 0, 6, 11, 79009, 0, 0 zio_data_buf_36864: 36864, 0, 871, 87, 10066, 0, 0 zio_buf_40960: 40960, 0, 8, 10, 80622, 0, 0 zio_data_buf_40960: 40960, 0, 785, 57, 8149, 0, 0 zio_buf_45056: 45056, 0, 50, 10, 70143, 0, 0 zio_data_buf_45056: 45056, 0, 659, 61, 6889, 0, 0 zio_buf_49152: 49152, 0, 190, 9, 73508, 0, 0 zio_data_buf_49152: 49152, 0, 512, 42, 8420, 0, 0 zio_buf_53248: 53248, 0, 25, 10, 99339, 0, 0 zio_data_buf_53248: 53248, 0, 465, 30, 4838, 0, 0 zio_buf_57344: 57344, 0, 2, 15, 95818, 0, 0 zio_data_buf_57344: 57344, 0, 337, 33, 4753, 0, 0 zio_buf_61440: 61440, 0, 1, 23, 98972, 0, 0 zio_data_buf_61440: 61440, 0, 294, 33, 3792, 0, 0 zio_buf_65536: 65536, 0, 1, 9, 114664, 0, 0 zio_data_buf_65536: 65536, 0, 291, 29, 5152, 0, 0 zio_buf_69632: 69632, 0, 2, 10, 90765, 0, 0 zio_data_buf_69632: 69632, 0, 291, 22, 3555, 0, 0 zio_buf_73728: 73728, 0, 1, 12, 87316, 0, 0 zio_data_buf_73728: 73728, 0, 219, 23, 3361, 0, 0 zio_buf_77824: 77824, 0, 2, 9, 101758, 0, 0 zio_data_buf_77824: 77824, 0, 196, 25, 2907, 0, 0 zio_buf_81920: 81920, 0, 0, 15, 129645, 0, 0 zio_data_buf_81920: 81920, 0, 187, 18, 3578, 0, 0 zio_buf_86016: 86016, 0, 1, 7, 99241, 0, 0 zio_data_buf_86016: 86016, 0, 180, 22, 26051, 0, 0 zio_buf_90112: 90112, 0, 1, 12, 51491, 0, 0 zio_data_buf_90112: 90112, 0, 163, 18, 2438, 0, 0 zio_buf_94208: 94208, 0, 0, 9, 51043, 0, 0 zio_data_buf_94208: 94208, 0, 149, 12, 2288, 0, 0 zio_buf_98304: 98304, 0, 2, 8, 50322, 0, 0 zio_data_buf_98304: 98304, 0, 170, 25, 3002, 0, 0 zio_buf_102400: 102400, 0, 2, 10, 44642, 0, 0 zio_data_buf_102400: 102400, 0, 145, 19, 2017, 0, 0 zio_buf_106496: 106496, 0, 0, 15, 161697, 0, 0 zio_data_buf_106496: 106496, 0, 161, 16, 1944, 0, 0 zio_buf_110592: 110592, 0, 0, 19, 146777, 0, 0 zio_data_buf_110592: 110592, 0, 97, 15, 1636, 0, 0 zio_buf_114688: 114688, 0, 2, 8, 51698, 0, 0 zio_data_buf_114688: 114688, 0, 115, 20, 2691, 0, 0 zio_buf_118784: 118784, 0, 0, 8, 62915, 0, 0 zio_data_buf_118784: 118784, 0, 96, 11, 1440, 0, 0 zio_buf_122880: 122880, 0, 0, 12, 85924, 0, 0 zio_data_buf_122880: 122880, 0, 73, 17, 1510, 0, 0 zio_buf_126976: 126976, 0, 1, 11, 86133, 0, 0 zio_data_buf_126976: 126976, 0, 80, 9, 1351, 0, 0 zio_buf_131072: 131072, 0, 43, 43, 548063, 0, 0 zio_data_buf_131072: 131072, 0, 67545, 2, 646567, 0, 0 lz4_ctx: 16384, 0, 0, 16, 2249605, 0, 0 sa_cache: 80, 0, 109455, 66445, 2663034, 0, 0 dnode_t: 744, 0, 501793, 259152, 873895, 0, 0 dmu_buf_impl_t: 224, 0, 635273, 343604, 3042828, 0, 0 arc_buf_hdr_t: 216, 0, 643049, 141229, 3158205, 0, 0 arc_buf_t: 72, 0, 481443, 119872, 3705908, 0, 0 zil_lwb_cache: 192, 0, 8, 5072, 66838, 0, 0 zfs_znode_cache: 368, 0, 109455, 66105, 2663034, 0, 0 pipe: 744, 0, 113, 177, 66135, 0, 0 procdesc: 128, 0, 0, 0, 0, 0, 0 ksiginfo: 112, 0, 237, 2528, 8311580, 0, 0 itimer: 352, 0, 1, 32, 1, 0, 0 KNOTE: 128, 0, 376, 1391,17307485, 0, 0 socket: 696, 523340, 258, 302, 668458, 0, 0 unpcb: 240, 523344, 151, 649, 130961, 0, 0 ipq: 56, 31879, 0, 2059, 317, 0, 0 udp_inpcb: 392, 523340, 19, 421, 469503, 0, 0 udpcb: 16, 523586, 19, 2993, 469503, 0, 0 tcp_inpcb: 392, 523340, 100, 370, 67983, 0, 0 tcpcb: 1024, 523340, 78, 234, 67983, 0, 0 tcptw: 88, 27810, 22, 1013, 22066, 0, 0 syncache: 160, 15375, 0, 675, 45270, 0, 0 hostcache: 136, 15370, 255, 499, 3892, 0, 0 tcpreass: 40, 63700, 0, 2600, 7616, 0, 0 sackhole: 32, 0, 0, 1875, 7184, 0, 0 sctp_ep: 1408, 523340, 0, 0, 0, 0, 0 sctp_asoc: 2352, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80012, 0, 1328, 8, 0, 0 sctp_raddr: 728, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400026, 0, 0, 0, 0, 0 sctp_readq: 104, 400026, 0, 0, 0, 0, 0 sctp_stream_msg_out: 104, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400000, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 udplite_inpcb: 392, 523340, 0, 0, 0, 0, 0 ripcb: 392, 523340, 2, 58, 3, 0, 0 rtentry: 200, 0, 32, 408, 34, 0, 0 selfd: 56, 0, 333, 2223,29658712, 0, 0 SWAPMETA: 288, 2037438, 0, 0, 0, 0, 0 thebighonker.lerctr.org /home/ler -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d5bda7f9ad7f86ec35ab6ba5571aa8bc>