From owner-freebsd-stable@FreeBSD.ORG Tue Oct 11 16:53:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8637C106564A; Tue, 11 Oct 2011 16:53:21 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 15D758FC12; Tue, 11 Oct 2011 16:53:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:CC:To:Date:From:Subject:Content-Type:MIME-Version:In-Reply-To:References; bh=T/9Dk+rGNqACVNgs8K3vCzMRTK0q26PKHyYLqQJdUDY=; b=qh2a8ZGbxz6xLEkXC/igW7slhqhBvJxrJjpZ178L3Tbcqt7Ox4r8/UFZyx+VgXp+otn6RGfnwTazAUDIjhDOPzXPqRp/rAykGNmaea88XxmKhBCVoeAbuRtm2Z4mFyLfMvYWuKG/aAMPdoIh9hTUN4y+NUiBIFncsJD4KCAd4A4=; Received: from [32.97.110.64] (port=35221 helo=Android-A100001B9859B8.austin.ibm.com) by thebighonker.lerctr.org with esmtpa (Exim 4.76 (FreeBSD)) (envelope-from ) id 1RDfZp-000GmI-UR; Tue, 11 Oct 2011 11:53:18 -0500 References: <69ffedd5-d750-44f4-b934-61a4abe3c59c@email.android.com> User-Agent: K-9 Mail for Android In-Reply-To: <69ffedd5-d750-44f4-b934-61a4abe3c59c@email.android.com> MIME-Version: 1.0 From: Larry Rosenman Date: Tue, 11 Oct 2011 11:53:16 -0500 To: Jack Vogel Message-ID: <6590d023-cd29-4d98-a3b5-161aa14bd006@email.android.com> X-Spam-Score: 0.0 (/) X-LERCTR-Spam-Score: 0.0 (/) X-Spam-Report: SpamScore (0.0/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, FM_MULTI_ODD2=1.1, HTML_MESSAGE=0.001, NORMAL_HTTP_TO_IP=0.001, SARE_HTML_USL_OBFU=1.666, TW_TD=0.077, TW_XF=0.077 X-LERCTR-Spam-Report: SpamScore (0.0/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, FM_MULTI_ODD2=1.1, HTML_MESSAGE=0.001, NORMAL_HTTP_TO_IP=0.001, SARE_HTML_USL_OBFU=1.666, TW_TD=0.077, TW_XF=0.077 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , Jeremy Chadwick , John Baldwin Subject: Re: rsync corrupted MAC X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Oct 2011 16:53:21 -0000 Not sure when it broke. I rebuilt the 9.0 server as 9.0, and ran the script= and it started giving this. -- Sent from my Android phone with K-9 Mail.= Please excuse my brevity. Jack Vogel wrote: Oh, I se= e. So, did you have a previous working state? Jack On Tue, Oct 11, 2011= at 12:06 AM, Larry Rosenman wrote: They are not local to= each other. See the diagram. They are across the internet from each other.= -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. = Jack Vogel wrote: Well, for a start I'd get both inte= rfaces at the same speed, sounds like a hardware issue of some sort, cable = or switch maybe? Jack On Mon, Oct 10, 2011 at 5:42 PM, Larry Rosenman wrote: On Mon, 10 Oct 2011, Jeremy Chadwick wrote: On Mon,= Oct 10, 2011 at 04:15:25PM -0500, Larry Rosenman wrote: On 10/10/2011 3:5= 7 PM, Louis Mamakos wrote: On Oct 10, 2011, at 2:38 PM, Larry Rosenman wro= te: On 10/10/2011 10:47 AM, John Baldwin wrote: On Sunday, October 09, 20= 11 5:06:26 pm Larry Rosenman wrote: Any ideas on which side or what might = be broke here? ler/MAIL-ARCHIVE/2008/12/INBOX Corrupted MAC on input. Disc= onnecting: Packet corrupt rsync: connection unexpectedly closed (33845045 b= ytes received so far) [receiver] rsync error: error in rsync protocol dat= a stream (code 12) at io.c(605) [receiver=3D3.0.9] rsync: connection unex= pectedly closed (1450 bytes received so far) [generator] rsync error: une= xplained error (code 255) at io.c(605) [generator=3D3.0.9] I've had somewh= at similar issues (ssh getting corruption in its data stream) when a NIC in= my netbook was corrupting packet data when it ran at 1G (it worked fine at= 10/100). Pyun eventually fixed the issue by applying enough workarounds (= it was likely a hardware bug in the NIC's chipset). However, it wasn't eas= y to debug unfortunately. :( Any ideas on where to start? from the 8.2 bo= x (tbh.lerctr.org in the script): 8.2->PIX->Provider->Internet->Motorola S= BG6580 (Time-Warner)->Trendnet TEG-160WS Gig switch->9.0 box (borg.lerctr.o= rg). So, where do I start? I'd turn off IP / TCP / UDP checksum offloadin= g on your NIC if it supports it, and see if you are getting network layer c= hecksum errors. If the IP checksum is wrong, then it happened on the last = hops between the NIC and memory or across the previous network hop. Good= idea, but, it didn't show ANY errors on EITHER side (both are em nics). N= ext? $ ifconfig em0 em0: flags=3D8843 metric 0 mtu 1500 options=3D2098 ether 00:30:48:2e:99:ba inet 192.147.25.65 net= mask 0xffffff00 broadcast 192.147.25.255 inet6 fe80::230:48ff:fe2e:9= 9ba%em0 prefixlen 64 scopeid 0x1 inet 192.147.25.45 netmask 0xffffff= 00 broadcast 192.147.25.255 inet 192.147.25.11 netmask 0xffffff00 br= oadcast 192.147.25.255 nd6 options=3D3 = media: Ethernet autoselect (100baseTX ) status: acti= ve $ $ uname -a FreeBSD thebighonker.lerctr.org 8.2-STABLE FreeBSD 8.2-STAB= LE #45: Sat Oct 8 10:57:43 CDT 2011 root@thebighonker.lerctr.org:/usr/obj/= usr/src/sys/THEBIGHONKER amd64 $ $ ifconfig em0 em0: flags=3D8843 metric 0 mtu 1500 options=3D2088 ether 00:30:48:8e:9f:f3 inet 1= 92.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255 inet6 fe80= ::230:48ff:fe8e:9ff3%em0 prefixlen 64 scopeid 0x1 nd6 options=3D29 media: Ethernet autoselect (100= 0baseT ) status: active $ uname -a FreeBSD borg.lerctr.= org 9.0-BETA3 FreeBSD 9.0-BETA3 #1: Sun Oct 9 10:03:42 CDT 2011 root@borg.= lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 $ Can you please provi= de output from the following commands executed on the machine showing the p= roblem? The above commands show nothing useful, other than the fact that o= ne machine is at 100/full and the other is at 1000/full (I don't know your = network setup). Commands: * netstat -inbd -I em0 * sysctl -a dev.em.0 * I= ssue command "sysctl dev.em.0.debug=3D1", then type "dmesg" and provide al= l of the new output you will see at the bottom that pertains to the NIC I= f you Google this problem, you will find that the majority of the time it's= caused by NIC drivers acting oddly. Also, I believe the em(4) driver in 9= .x is slightly different than on 8.x, so I'm CC'ing Jack Vogel here. fro= m 9.0: Name Mtu Network Address Ipkts Ierrs Idrop = Ibytes Opkts Oerrs Obytes Coll Drop em0 1500 00:= 30:48:8e:9f:f3 69776975 0 0 59660392277 52592789 0 104743924118= 0 0 em0 1500 192.168.200.0 192.168.200.4 69759773 - = - 58681934612 96397272 - 104003761109 - - em0 1500 fe80::230:= 48f fe80::230:48ff:fe 0 - - 0 3 - = 248 - - dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.= 3 dev.em.0.%driver: em dev.em.0.%location: slot=3D0 function=3D0 dev.em.0.%= pnpinfo: vendor=3D0x8086 device=3D0x1096 subvendor=3D0x15d9 subdevice=3D0x0= 000 class=3D0x020000 dev.em.0.%parent: pci6 dev.em.0.nvm: -1 dev.em.0.debug= : -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int= _delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 = dev.em.0.flow_control: 3 dev.em.0.eee_control: 0 dev.em.0.link_irq: 0 dev.e= m.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 d= ev.em.0.tx_dma_fail: 21755 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeou= ts: 0 dev.em.0.device_control: 1851969 dev.em.0.rx_control: 67141634 dev.em= .0.fc_high_water: 30720 dev.em.0.fc_low_water: 29220 dev.em.0.queue0.txd_he= ad: 136 dev.em.0.queue0.txd_tail: 136 dev.em.0.queue0.tx_irq: 0 dev.em.0.qu= eue0.no_desc_avail: 0 dev.em.0.queue0.rxd_head: 2 dev.em.0.queue0.rxd_tail:= 1 dev.em.0.queue0.rx_irq: 0 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac= _stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stat= s.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.sym= bol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defe= r_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_= buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragme= nted: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber:= 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.= mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.ma= c_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_= recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd= : 69774324 dev.em.0.mac_stats.good_pkts_recvd: 69774324 dev.em.0.mac_stats.= bcast_pkts_recvd: 28156 dev.em.0.mac_stats.mcast_pkts_recvd: 1758 dev.em.0.= mac_stats.rx_frames_64: 54177 dev.em.0.mac_stats.rx_frames_65_127: 30157358= dev.em.0.mac_stats.rx_frames_128_255: 1092948 dev.em.0.mac_stats.rx_frames= _256_511: 125295 dev.em.0.mac_stats.rx_frames_512_1023: 128081 dev.em.0.mac= _stats.rx_frames_1024_1522: 38216465 dev.em.0.mac_stats.good_octets_recvd: = 59938624047 dev.em.0.mac_stats.good_octets_txd: 106613836902 dev.em.0.mac_s= tats.total_pkts_txd: 96250538 dev.em.0.mac_stats.good_pkts_txd: 96250538 de= v.em.0.mac_stats.bcast_pkts_txd: 2989 dev.em.0.mac_stats.mcast_pkts_txd: 0 = dev.em.0.mac_stats.tx_frames_64: 7551 dev.em.0.mac_stats.tx_frames_65_127: = 26727682 dev.em.0.mac_stats.tx_frames_128_255: 227574 dev.em.0.mac_stats.tx= _frames_256_511: 167383 dev.em.0.mac_stats.tx_frames_512_1023: 302141 dev.e= m.0.mac_stats.tx_frames_1024_1522: 68818207 dev.em.0.mac_stats.tso_txd: 172= 44234 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.interrupts.asserts: 65945= 396 dev.em.0.interrupts.rx_pkt_timer: 8917 dev.em.0.interrupts.rx_abs_timer= : 0 dev.em.0.interrupts.tx_pkt_timer: 1461 dev.em.0.interrupts.tx_abs_timer= : 1951 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_queue_m= in_thresh: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.= rx_overrun: 0 Interface is RUNNING and INACTIVE em0: hw tdh =3D 221, hw td= t =3D 221 em0: hw rdh =3D 467, hw rdt =3D 466 em0: Tx Queue Status =3D 0 em= 0: TX descriptors avail =3D 1024 em0: Tx Descriptors avail failure =3D 0 em= 0: RX discarded packets =3D 0 em0: RX Next to Check =3D 467 em0: RX Next to= Refresh =3D 466 $ from 8.2: Name Mtu Network Address = Ipkts Ierrs Idrop Ibytes Opkts Oerrs Obytes Coll Drop em0 = 1500 00:30:48:2e:99:ba 276150 0 0 42614583 285= 398 0 207023352 0 0 em0 1500 192.147.25.0/ 192.147.25.65 = 318062 - - 48331105 285363 - 203026287 - - em0 = 1500 fe80::230:48f fe80::230:48ff:fe 0 - - 0 = 1 - 96 - - em0 1500 192.147.25.0/ 192.147.25.45 = 25071 - - 1782211 0 - 0 - - em0 = 1500 192.147.25.0/ 192.147.25.11 38433 - - 2742827 = 0 - 0 - - dev.em.0.%desc: Intel(R) PRO/1000 Legac= y Network Connection 1.0.3 dev.em.0.%driver: em dev.em.0.%location: slot=3D= 2 function=3D0 dev.em.0.%pnpinfo: vendor=3D0x8086 device=3D0x1079 subvendor= =3D0x15d9 subdevice=3D0x117a class=3D0x020000 dev.em.0.%parent: pci3 dev.em= .0.nvm: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_a= bs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit= : 100 dev.em.0.flow_control: 3 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster= _alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.tx_desc= _fail1: 0 dev.em.0.tx_desc_fail2: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchd= og_timeouts: 0 dev.em.0.device_control: 1089471041 dev.em.0.rx_control: 327= 70 dev.em.0.fc_high_water: 47104 dev.em.0.fc_low_water: 45604 dev.em.0.fifo= _workaround: 0 dev.em.0.fifo_reset: 0 dev.em.0.txd_head: 73 dev.em.0.txd_ta= il: 75 dev.em.0.rxd_head: 130 dev.em.0.rxd_tail: 129 dev.em.0.mac_stats.exc= ess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_c= oll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: = 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0= dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev= .em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em= .0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.= 0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_sta= ts.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll= _ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 = dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.ma= c_stats.total_pkts_recvd: 276318 dev.em.0.mac_stats.good_pkts_recvd: 276318= dev.em.0.mac_stats.bcast_pkts_recvd: 8 dev.em.0.mac_stats.mcast_pkts_recvd= : 0 dev.em.0.mac_stats.rx_frames_64: 9012 dev.em.0.mac_stats.rx_frames_65_1= 27: 205540 dev.em.0.mac_stats.rx_frames_128_255: 44078 dev.em.0.mac_stats.r= x_frames_256_511: 3166 dev.em.0.mac_stats.rx_frames_512_1023: 3934 dev.em.0= .mac_stats.rx_frames_1024_1522: 10588 dev.em.0.mac_stats.good_octets_recvd:= 43761022 dev.em.0.mac_stats.good_octets_txd: 208238998 dev.em.0.mac_stats.= total_pkts_txd: 285534 dev.em.0.mac_stats.good_pkts_txd: 285534 dev.em.0.ma= c_stats.bcast_pkts_txd: 22 dev.em.0.mac_stats.mcast_pkts_txd: 3 dev.em.0.ma= c_stats.tx_frames_64: 6172 dev.em.0.mac_stats.tx_frames_65_127: 75983 dev.e= m.0.mac_stats.tx_frames_128_255: 53030 dev.em.0.mac_stats.tx_frames_256_511= : 23216 dev.em.0.mac_stats.tx_frames_512_1023: 1472 dev.em.0.mac_stats.tx_f= rames_1024_1522: 125661 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.ts= o_ctx_fail: 0 $ sudo sysctl dev.em.0.debug=3D1 sysctl: unknown oid 'dev.em= .0.debug' $ -- Larry Rosenman http://www.lerctr.org/~= ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: = 430 Valona Loop, Round Rock, TX 78681-3893 ________________________________= _______________ freebsd-stable@freebsd.org mailing list http://lists.freebs= d.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "fre= ebsd-stable-unsubscribe@freebsd.org"