Date: Tue, 11 Oct 2011 09:37:55 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Larry Rosenman <ler@lerctr.org> Cc: freebsd-stable@freebsd.org, "Vogel, Jack" <jack.vogel@intel.com>, Jeremy Chadwick <freebsd@jdc.parodius.com>, John Baldwin <jhb@freebsd.org> Subject: Re: rsync corrupted MAC Message-ID: <CAFOYbc=kUq7pbaK%2BjPUG984h5uixddck-geZP3rehDpyjsEANg@mail.gmail.com> In-Reply-To: <25c4965e-fb4d-4991-ae7b-e4fe89741359@email.android.com> References: <b3b95bea-d2d9-48de-9bf3-206bfbfed158@email.android.com> <25c4965e-fb4d-4991-ae7b-e4fe89741359@email.android.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Oh, I see. So, did you have a previous working state? Jack On Tue, Oct 11, 2011 at 12:06 AM, Larry Rosenman <ler@lerctr.org> wrote: > ** They are not local to each other. See the diagram. They are across the > internet from each other. > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > > > Jack Vogel <jfvogel@gmail.com> wrote: >> >> Well, for a start I'd get both interfaces at the same speed, sounds like a >> hardware >> issue of some sort, cable or switch maybe? >> >> Jack >> >> >> On Mon, Oct 10, 2011 at 5:42 PM, Larry Rosenman <ler@lerctr.org> wrote: >> >>> On Mon, 10 Oct 2011, Jeremy Chadwick wrote: >>> >>> On Mon, Oct 10, 2011 at 04:15:25PM -0500, Larry Rosenman wrote: >>>> >>>>> On 10/10/2011 3:57 PM, Louis Mamakos wrote: >>>>> >>>>>> On Oct 10, 2011, at 2:38 PM, Larry Rosenman wrote: >>>>>> >>>>>> On 10/10/2011 10:47 AM, John Baldwin wrote: >>>>>>> >>>>>>>> On Sunday, October 09, 2011 5:06:26 pm Larry Rosenman wrote: >>>>>>>> >>>>>>>>> Any ideas on which side or what might be broke here? >>>>>>>>> >>>>>>>>> ler/MAIL-ARCHIVE/2008/12/INBOX >>>>>>>>> Corrupted MAC on input. >>>>>>>>> Disconnecting: Packet corrupt >>>>>>>>> rsync: connection unexpectedly closed (33845045 bytes received so >>>>>>>>> far) >>>>>>>>> >>>>>>>> [receiver] >>>>>>>> >>>>>>>>> rsync error: error in rsync protocol data stream (code 12) at >>>>>>>>> io.c(605) >>>>>>>>> >>>>>>>> [receiver=3.0.9] >>>>>>>> >>>>>>>>> rsync: connection unexpectedly closed (1450 bytes received so far) >>>>>>>>> >>>>>>>> [generator] >>>>>>>> >>>>>>>>> rsync error: unexplained error (code 255) at io.c(605) >>>>>>>>> [generator=3.0.9] >>>>>>>>> >>>>>>>> I've had somewhat similar issues (ssh getting corruption in its data >>>>>>>> stream) >>>>>>>> when a NIC in my netbook was corrupting packet data when it ran at >>>>>>>> 1G (it >>>>>>>> worked fine at 10/100). Pyun eventually fixed the issue by applying >>>>>>>> enough >>>>>>>> workarounds (it was likely a hardware bug in the NIC's chipset). >>>>>>>> However, it >>>>>>>> wasn't easy to debug unfortunately. :( >>>>>>>> >>>>>>>> Any ideas on where to start? >>>>>>> >>>>>>> from the 8.2 box (tbh.lerctr.org in the script): >>>>>>> >>>>>>> 8.2->PIX->Provider->Internet->**Motorola SBG6580 >>>>>>> (Time-Warner)->Trendnet TEG-160WS Gig switch->9.0 box ( >>>>>>> borg.lerctr.org). >>>>>>> >>>>>>> So, where do I start? >>>>>>> >>>>>> I'd turn off IP / TCP / UDP checksum offloading on your NIC if it >>>>>> supports it, and see if you are getting network layer checksum errors. If >>>>>> the IP checksum is wrong, then it happened on the last hops between the NIC >>>>>> and memory or across the previous network hop. >>>>>> >>>>>> >>>>>> >>>>>> Good idea, but, it didn't show ANY errors on EITHER side (both are >>>>> em nics). >>>>> >>>>> Next? >>>>> $ ifconfig em0 >>>>> em0: flags=8843<UP,BROADCAST,**RUNNING,SIMPLEX,MULTICAST> metric 0 mtu >>>>> 1500 >>>>> options=2098<VLAN_MTU,VLAN_**HWTAGGING,VLAN_HWCSUM,WOL_**MAGIC> >>>>> ether 00:30:48:2e:99:ba >>>>> inet 192.147.25.65 netmask 0xffffff00 broadcast 192.147.25.255 >>>>> inet6 fe80::230:48ff:fe2e:99ba%em0 prefixlen 64 scopeid 0x1 >>>>> inet 192.147.25.45 netmask 0xffffff00 broadcast 192.147.25.255 >>>>> inet 192.147.25.11 netmask 0xffffff00 broadcast 192.147.25.255 >>>>> nd6 options=3<PERFORMNUD,ACCEPT_**RTADV> >>>>> media: Ethernet autoselect (100baseTX <full-duplex>) >>>>> status: active >>>>> $ >>>>> $ uname -a >>>>> FreeBSD thebighonker.lerctr.org 8.2-STABLE FreeBSD 8.2-STABLE #45: >>>>> Sat Oct 8 10:57:43 CDT 2011 >>>>> root@thebighonker.lerctr.org:/**usr/obj/usr/src/sys/**THEBIGHONKER >>>>> amd64 >>>>> $ >>>>> >>>>> >>>>> >>>>> $ ifconfig em0 >>>>> em0: flags=8843<UP,BROADCAST,**RUNNING,SIMPLEX,MULTICAST> metric 0 mtu >>>>> 1500 >>>>> options=2088<VLAN_MTU,VLAN_**HWCSUM,WOL_MAGIC> >>>>> ether 00:30:48:8e:9f:f3 >>>>> inet 192.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255 >>>>> inet6 fe80::230:48ff:fe8e:9ff3%em0 prefixlen 64 scopeid 0x1 >>>>> nd6 options=29<PERFORMNUD,**IFDISABLED,AUTO_LINKLOCAL> >>>>> media: Ethernet autoselect (1000baseT <full-duplex>) >>>>> status: active >>>>> $ uname -a >>>>> FreeBSD borg.lerctr.org 9.0-BETA3 FreeBSD 9.0-BETA3 #1: Sun Oct 9 >>>>> 10:03:42 CDT 2011 >>>>> root@borg.lerctr.org:/usr/obj/**usr/src/sys/BORG-DTRACE amd64 >>>>> $ >>>>> >>>> >>>> Can you please provide output from the following commands executed on >>>> the machine showing the problem? The above commands show nothing >>>> useful, other than the fact that one machine is at 100/full and the >>>> other is at 1000/full (I don't know your network setup). Commands: >>>> >>>> * netstat -inbd -I em0 >>>> * sysctl -a dev.em.0 >>>> * Issue command "sysctl dev.em.0.debug=1", then type "dmesg" and >>>> provide all of the new output you will see at the bottom that >>>> pertains to the NIC >>>> >>>> If you Google this problem, you will find that the majority of the time >>>> it's caused by NIC drivers acting oddly. >>>> >>>> Also, I believe the em(4) driver in 9.x is slightly different than on >>>> 8.x, so I'm CC'ing Jack Vogel here. >>>> >>>> >>>> >>> from 9.0: >>> >>> Name Mtu Network Address Ipkts Ierrs Idrop >>> Ibytes Opkts Oerrs Obytes Coll Drop >>> em0 1500 <Link#1> 00:30:48:8e:9f:f3 69776975 0 0 >>> 59660392277 52592789 0 104743924118 0 0 em0 1500 192.168.200.0 >>> 192.168.200.4 69759773 - - 58681934612 96397272 - >>> 104003761109 - - em0 1500 fe80::230:48f fe80::230:48ff:fe 0 >>> - - 0 3 - 248 - - >>> >>> >>> dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 >>> dev.em.0.%driver: em >>> dev.em.0.%location: slot=0 function=0 >>> dev.em.0.%pnpinfo: vendor=0x8086 device=0x1096 subvendor=0x15d9 >>> subdevice=0x0000 class=0x020000 >>> dev.em.0.%parent: pci6 >>> dev.em.0.nvm: -1 >>> dev.em.0.debug: -1 >>> dev.em.0.rx_int_delay: 0 >>> dev.em.0.tx_int_delay: 66 >>> dev.em.0.rx_abs_int_delay: 66 >>> dev.em.0.tx_abs_int_delay: 66 >>> dev.em.0.rx_processing_limit: 100 >>> dev.em.0.flow_control: 3 >>> dev.em.0.eee_control: 0 >>> dev.em.0.link_irq: 0 >>> dev.em.0.mbuf_alloc_fail: 0 >>> dev.em.0.cluster_alloc_fail: 0 >>> dev.em.0.dropped: 0 >>> dev.em.0.tx_dma_fail: 21755 >>> dev.em.0.rx_overruns: 0 >>> dev.em.0.watchdog_timeouts: 0 >>> dev.em.0.device_control: 1851969 >>> dev.em.0.rx_control: 67141634 >>> dev.em.0.fc_high_water: 30720 >>> dev.em.0.fc_low_water: 29220 >>> dev.em.0.queue0.txd_head: 136 >>> dev.em.0.queue0.txd_tail: 136 >>> dev.em.0.queue0.tx_irq: 0 >>> dev.em.0.queue0.no_desc_avail: 0 >>> dev.em.0.queue0.rxd_head: 2 >>> dev.em.0.queue0.rxd_tail: 1 >>> dev.em.0.queue0.rx_irq: 0 >>> dev.em.0.mac_stats.excess_**coll: 0 >>> dev.em.0.mac_stats.single_**coll: 0 >>> dev.em.0.mac_stats.multiple_**coll: 0 >>> dev.em.0.mac_stats.late_coll: 0 >>> dev.em.0.mac_stats.collision_**count: 0 >>> dev.em.0.mac_stats.symbol_**errors: 0 >>> dev.em.0.mac_stats.sequence_**errors: 0 >>> dev.em.0.mac_stats.defer_**count: 0 >>> dev.em.0.mac_stats.missed_**packets: 0 >>> dev.em.0.mac_stats.recv_no_**buff: 0 >>> dev.em.0.mac_stats.recv_**undersize: 0 >>> dev.em.0.mac_stats.recv_**fragmented: 0 >>> dev.em.0.mac_stats.recv_**oversize: 0 >>> dev.em.0.mac_stats.recv_**jabber: 0 >>> dev.em.0.mac_stats.recv_errs: 0 >>> dev.em.0.mac_stats.crc_errs: 0 >>> dev.em.0.mac_stats.alignment_**errs: 0 >>> dev.em.0.mac_stats.coll_ext_**errs: 0 >>> dev.em.0.mac_stats.xon_recvd: 0 >>> dev.em.0.mac_stats.xon_txd: 0 >>> dev.em.0.mac_stats.xoff_recvd: 0 >>> dev.em.0.mac_stats.xoff_txd: 0 >>> dev.em.0.mac_stats.total_pkts_**recvd: 69774324 >>> dev.em.0.mac_stats.good_pkts_**recvd: 69774324 >>> dev.em.0.mac_stats.bcast_pkts_**recvd: 28156 >>> dev.em.0.mac_stats.mcast_pkts_**recvd: 1758 >>> dev.em.0.mac_stats.rx_frames_**64: 54177 >>> dev.em.0.mac_stats.rx_frames_**65_127: 30157358 >>> dev.em.0.mac_stats.rx_frames_**128_255: 1092948 >>> dev.em.0.mac_stats.rx_frames_**256_511: 125295 >>> dev.em.0.mac_stats.rx_frames_**512_1023: 128081 >>> dev.em.0.mac_stats.rx_frames_**1024_1522: 38216465 >>> dev.em.0.mac_stats.good_**octets_recvd: 59938624047 >>> dev.em.0.mac_stats.good_**octets_txd: 106613836902 >>> dev.em.0.mac_stats.total_pkts_**txd: 96250538 >>> dev.em.0.mac_stats.good_pkts_**txd: 96250538 >>> dev.em.0.mac_stats.bcast_pkts_**txd: 2989 >>> dev.em.0.mac_stats.mcast_pkts_**txd: 0 >>> dev.em.0.mac_stats.tx_frames_**64: 7551 >>> dev.em.0.mac_stats.tx_frames_**65_127: 26727682 >>> dev.em.0.mac_stats.tx_frames_**128_255: 227574 >>> dev.em.0.mac_stats.tx_frames_**256_511: 167383 >>> dev.em.0.mac_stats.tx_frames_**512_1023: 302141 >>> dev.em.0.mac_stats.tx_frames_**1024_1522: 68818207 >>> dev.em.0.mac_stats.tso_txd: 17244234 >>> dev.em.0.mac_stats.tso_ctx_**fail: 0 >>> dev.em.0.interrupts.asserts: 65945396 >>> dev.em.0.interrupts.rx_pkt_**timer: 8917 >>> dev.em.0.interrupts.rx_abs_**timer: 0 >>> dev.em.0.interrupts.tx_pkt_**timer: 1461 >>> dev.em.0.interrupts.tx_abs_**timer: 1951 >>> dev.em.0.interrupts.tx_queue_**empty: 0 >>> dev.em.0.interrupts.tx_queue_**min_thresh: 0 >>> dev.em.0.interrupts.rx_desc_**min_thresh: 0 >>> dev.em.0.interrupts.rx_**overrun: 0 >>> >>> Interface is RUNNING and INACTIVE >>> em0: hw tdh = 221, hw tdt = 221 >>> em0: hw rdh = 467, hw rdt = 466 >>> em0: Tx Queue Status = 0 >>> em0: TX descriptors avail = 1024 >>> em0: Tx Descriptors avail failure = 0 >>> em0: RX discarded packets = 0 >>> em0: RX Next to Check = 467 >>> em0: RX Next to Refresh = 466 >>> $ >>> >>> from 8.2: >>> >>> Name Mtu Network Address Ipkts Ierrs Idrop >>> Ibytes Opkts Oerrs Obytes Coll Drop >>> em0 1500 <Link#1> 00:30:48:2e:99:ba 276150 0 0 >>> 42614583 285398 0 207023352 0 0 em0 1500 192.147.25.0/192.147.25.65 318062 - - 48331105 285363 - 203026287 >>> - - em0 1500 fe80::230:48f fe80::230:48ff:fe 0 - - >>> 0 1 - 96 - - em0 1500 >>> 192.147.25.0/ 192.147.25.45 25071 - - 1782211 0 >>> - 0 - - em0 1500 192.147.25.0/ 192.147.25.11 >>> 38433 - - 2742827 0 - 0 - - >>> >>> >>> dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.3 >>> dev.em.0.%driver: em >>> dev.em.0.%location: slot=2 function=0 >>> dev.em.0.%pnpinfo: vendor=0x8086 device=0x1079 subvendor=0x15d9 >>> subdevice=0x117a class=0x020000 >>> dev.em.0.%parent: pci3 >>> dev.em.0.nvm: -1 >>> dev.em.0.rx_int_delay: 0 >>> dev.em.0.tx_int_delay: 66 >>> dev.em.0.rx_abs_int_delay: 66 >>> dev.em.0.tx_abs_int_delay: 66 >>> dev.em.0.rx_processing_limit: 100 >>> dev.em.0.flow_control: 3 >>> dev.em.0.mbuf_alloc_fail: 0 >>> dev.em.0.cluster_alloc_fail: 0 >>> dev.em.0.dropped: 0 >>> dev.em.0.tx_dma_fail: 0 >>> dev.em.0.tx_desc_fail1: 0 >>> dev.em.0.tx_desc_fail2: 0 >>> dev.em.0.rx_overruns: 0 >>> dev.em.0.watchdog_timeouts: 0 >>> dev.em.0.device_control: 1089471041 >>> dev.em.0.rx_control: 32770 >>> dev.em.0.fc_high_water: 47104 >>> dev.em.0.fc_low_water: 45604 >>> dev.em.0.fifo_workaround: 0 >>> dev.em.0.fifo_reset: 0 >>> dev.em.0.txd_head: 73 >>> dev.em.0.txd_tail: 75 >>> dev.em.0.rxd_head: 130 >>> dev.em.0.rxd_tail: 129 >>> dev.em.0.mac_stats.excess_**coll: 0 >>> dev.em.0.mac_stats.single_**coll: 0 >>> dev.em.0.mac_stats.multiple_**coll: 0 >>> dev.em.0.mac_stats.late_coll: 0 >>> dev.em.0.mac_stats.collision_**count: 0 >>> dev.em.0.mac_stats.symbol_**errors: 0 >>> dev.em.0.mac_stats.sequence_**errors: 0 >>> dev.em.0.mac_stats.defer_**count: 0 >>> dev.em.0.mac_stats.missed_**packets: 0 >>> dev.em.0.mac_stats.recv_no_**buff: 0 >>> dev.em.0.mac_stats.recv_**undersize: 0 >>> dev.em.0.mac_stats.recv_**fragmented: 0 >>> dev.em.0.mac_stats.recv_**oversize: 0 >>> dev.em.0.mac_stats.recv_**jabber: 0 >>> dev.em.0.mac_stats.recv_errs: 0 >>> dev.em.0.mac_stats.crc_errs: 0 >>> dev.em.0.mac_stats.alignment_**errs: 0 >>> dev.em.0.mac_stats.coll_ext_**errs: 0 >>> dev.em.0.mac_stats.xon_recvd: 0 >>> dev.em.0.mac_stats.xon_txd: 0 >>> dev.em.0.mac_stats.xoff_recvd: 0 >>> dev.em.0.mac_stats.xoff_txd: 0 >>> dev.em.0.mac_stats.total_pkts_**recvd: 276318 >>> dev.em.0.mac_stats.good_pkts_**recvd: 276318 >>> dev.em.0.mac_stats.bcast_pkts_**recvd: 8 >>> dev.em.0.mac_stats.mcast_pkts_**recvd: 0 >>> dev.em.0.mac_stats.rx_frames_**64: 9012 >>> dev.em.0.mac_stats.rx_frames_**65_127: 205540 >>> dev.em.0.mac_stats.rx_frames_**128_255: 44078 >>> dev.em.0.mac_stats.rx_frames_**256_511: 3166 >>> dev.em.0.mac_stats.rx_frames_**512_1023: 3934 >>> dev.em.0.mac_stats.rx_frames_**1024_1522: 10588 >>> dev.em.0.mac_stats.good_**octets_recvd: 43761022 >>> dev.em.0.mac_stats.good_**octets_txd: 208238998 >>> dev.em.0.mac_stats.total_pkts_**txd: 285534 >>> dev.em.0.mac_stats.good_pkts_**txd: 285534 >>> dev.em.0.mac_stats.bcast_pkts_**txd: 22 >>> dev.em.0.mac_stats.mcast_pkts_**txd: 3 >>> dev.em.0.mac_stats.tx_frames_**64: 6172 >>> dev.em.0.mac_stats.tx_frames_**65_127: 75983 >>> dev.em.0.mac_stats.tx_frames_**128_255: 53030 >>> dev.em.0.mac_stats.tx_frames_**256_511: 23216 >>> dev.em.0.mac_stats.tx_frames_**512_1023: 1472 >>> dev.em.0.mac_stats.tx_frames_**1024_1522: 125661 >>> dev.em.0.mac_stats.tso_txd: 0 >>> dev.em.0.mac_stats.tso_ctx_**fail: 0 >>> >>> $ sudo sysctl dev.em.0.debug=1 >>> sysctl: unknown oid 'dev.em.0.debug' >>> $ >>> >>> -- >>> Larry Rosenman http://www.lerctr.org/~ler >>> Phone: +1 512-248-2683 E-Mail: ler@lerctr.org >>> US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 >>> ______________________________**_________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/**mailman/listinfo/freebsd-**stable<http://lists.freebsd.org/mailman/listinfo/freebsd-stable> >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@** >>> freebsd.org <freebsd-stable-unsubscribe@freebsd.org>" >>> >> >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbc=kUq7pbaK%2BjPUG984h5uixddck-geZP3rehDpyjsEANg>