From owner-freebsd-stable@FreeBSD.ORG Tue Oct 11 04:18:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C6831065674 for ; Tue, 11 Oct 2011 04:18:48 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6EC1E8FC14 for ; Tue, 11 Oct 2011 04:18:47 +0000 (UTC) Received: by wyj26 with SMTP id 26so10264251wyj.13 for ; Mon, 10 Oct 2011 21:18:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=MuyNVcbCGHiCzrCOT7Blrkp8rR6LrhI6dEKA52tbdpc=; b=k+kJur1SjgOfz6SDgsWxlhUkf+dImaX3r2b1//ojPU93fv735LZ10kCxT69FOQZfpQ 8x/Y3Mp8I0gjKhDAx9rkLX+quGW0NRhQzuZWn6BFm6koxLTyck2ignLQIC8CgQ6jsLAL cjELybYMVa73KxpmWuVsh9VaW37mom3Z1e3xI= MIME-Version: 1.0 Received: by 10.216.139.135 with SMTP id c7mr192334wej.28.1318305232941; Mon, 10 Oct 2011 20:53:52 -0700 (PDT) Received: by 10.180.106.35 with HTTP; Mon, 10 Oct 2011 20:53:52 -0700 (PDT) In-Reply-To: References: <201110101147.30558.jhb@freebsd.org> <4E933BBF.6070209@lerctr.org> <36C97D31-5D01-4AC2-8E48-9A8B04B98F91@transsys.com> <4E93606D.8070306@lerctr.org> <20111010230207.GA85243@icarus.home.lan> Date: Mon, 10 Oct 2011 20:53:52 -0700 Message-ID: From: Jack Vogel To: Larry Rosenman Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , Jeremy Chadwick , John Baldwin Subject: Re: rsync corrupted MAC X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Oct 2011 04:18:48 -0000 Well, for a start I'd get both interfaces at the same speed, sounds like a hardware issue of some sort, cable or switch maybe? Jack On Mon, Oct 10, 2011 at 5:42 PM, Larry Rosenman wrote: > On Mon, 10 Oct 2011, Jeremy Chadwick wrote: > > On Mon, Oct 10, 2011 at 04:15:25PM -0500, Larry Rosenman wrote: >> >>> On 10/10/2011 3:57 PM, Louis Mamakos wrote: >>> >>>> On Oct 10, 2011, at 2:38 PM, Larry Rosenman wrote: >>>> >>>> On 10/10/2011 10:47 AM, John Baldwin wrote: >>>>> >>>>>> On Sunday, October 09, 2011 5:06:26 pm Larry Rosenman wrote: >>>>>> >>>>>>> Any ideas on which side or what might be broke here? >>>>>>> >>>>>>> ler/MAIL-ARCHIVE/2008/12/INBOX >>>>>>> Corrupted MAC on input. >>>>>>> Disconnecting: Packet corrupt >>>>>>> rsync: connection unexpectedly closed (33845045 bytes received so >>>>>>> far) >>>>>>> >>>>>> [receiver] >>>>>> >>>>>>> rsync error: error in rsync protocol data stream (code 12) at >>>>>>> io.c(605) >>>>>>> >>>>>> [receiver=3.0.9] >>>>>> >>>>>>> rsync: connection unexpectedly closed (1450 bytes received so far) >>>>>>> >>>>>> [generator] >>>>>> >>>>>>> rsync error: unexplained error (code 255) at io.c(605) >>>>>>> [generator=3.0.9] >>>>>>> >>>>>> I've had somewhat similar issues (ssh getting corruption in its data >>>>>> stream) >>>>>> when a NIC in my netbook was corrupting packet data when it ran at 1G >>>>>> (it >>>>>> worked fine at 10/100). Pyun eventually fixed the issue by applying >>>>>> enough >>>>>> workarounds (it was likely a hardware bug in the NIC's chipset). >>>>>> However, it >>>>>> wasn't easy to debug unfortunately. :( >>>>>> >>>>>> Any ideas on where to start? >>>>> >>>>> from the 8.2 box (tbh.lerctr.org in the script): >>>>> >>>>> 8.2->PIX->Provider->Internet->**Motorola SBG6580 >>>>> (Time-Warner)->Trendnet TEG-160WS Gig switch->9.0 box (borg.lerctr.org >>>>> ). >>>>> >>>>> So, where do I start? >>>>> >>>> I'd turn off IP / TCP / UDP checksum offloading on your NIC if it >>>> supports it, and see if you are getting network layer checksum errors. If >>>> the IP checksum is wrong, then it happened on the last hops between the NIC >>>> and memory or across the previous network hop. >>>> >>>> >>>> >>>> Good idea, but, it didn't show ANY errors on EITHER side (both are >>> em nics). >>> >>> Next? >>> $ ifconfig em0 >>> em0: flags=8843 metric 0 mtu >>> 1500 >>> options=2098 >>> ether 00:30:48:2e:99:ba >>> inet 192.147.25.65 netmask 0xffffff00 broadcast 192.147.25.255 >>> inet6 fe80::230:48ff:fe2e:99ba%em0 prefixlen 64 scopeid 0x1 >>> inet 192.147.25.45 netmask 0xffffff00 broadcast 192.147.25.255 >>> inet 192.147.25.11 netmask 0xffffff00 broadcast 192.147.25.255 >>> nd6 options=3 >>> media: Ethernet autoselect (100baseTX ) >>> status: active >>> $ >>> $ uname -a >>> FreeBSD thebighonker.lerctr.org 8.2-STABLE FreeBSD 8.2-STABLE #45: >>> Sat Oct 8 10:57:43 CDT 2011 >>> root@thebighonker.lerctr.org:/**usr/obj/usr/src/sys/**THEBIGHONKER >>> amd64 >>> $ >>> >>> >>> >>> $ ifconfig em0 >>> em0: flags=8843 metric 0 mtu >>> 1500 >>> options=2088 >>> ether 00:30:48:8e:9f:f3 >>> inet 192.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255 >>> inet6 fe80::230:48ff:fe8e:9ff3%em0 prefixlen 64 scopeid 0x1 >>> nd6 options=29 >>> media: Ethernet autoselect (1000baseT ) >>> status: active >>> $ uname -a >>> FreeBSD borg.lerctr.org 9.0-BETA3 FreeBSD 9.0-BETA3 #1: Sun Oct 9 >>> 10:03:42 CDT 2011 >>> root@borg.lerctr.org:/usr/obj/**usr/src/sys/BORG-DTRACE amd64 >>> $ >>> >> >> Can you please provide output from the following commands executed on >> the machine showing the problem? The above commands show nothing >> useful, other than the fact that one machine is at 100/full and the >> other is at 1000/full (I don't know your network setup). Commands: >> >> * netstat -inbd -I em0 >> * sysctl -a dev.em.0 >> * Issue command "sysctl dev.em.0.debug=1", then type "dmesg" and >> provide all of the new output you will see at the bottom that >> pertains to the NIC >> >> If you Google this problem, you will find that the majority of the time >> it's caused by NIC drivers acting oddly. >> >> Also, I believe the em(4) driver in 9.x is slightly different than on >> 8.x, so I'm CC'ing Jack Vogel here. >> >> >> > from 9.0: > > Name Mtu Network Address Ipkts Ierrs Idrop Ibytes > Opkts Oerrs Obytes Coll Drop > em0 1500 00:30:48:8e:9f:f3 69776975 0 0 > 59660392277 52592789 0 104743924118 0 0 em0 1500 192.168.200.0 > 192.168.200.4 69759773 - - 58681934612 96397272 - > 104003761109 - - em0 1500 fe80::230:48f fe80::230:48ff:fe 0 > - - 0 3 - 248 - - > > > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=0 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x1096 subvendor=0x15d9 > subdevice=0x0000 class=0x020000 > dev.em.0.%parent: pci6 > dev.em.0.nvm: -1 > dev.em.0.debug: -1 > dev.em.0.rx_int_delay: 0 > dev.em.0.tx_int_delay: 66 > dev.em.0.rx_abs_int_delay: 66 > dev.em.0.tx_abs_int_delay: 66 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 3 > dev.em.0.eee_control: 0 > dev.em.0.link_irq: 0 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 21755 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1851969 > dev.em.0.rx_control: 67141634 > dev.em.0.fc_high_water: 30720 > dev.em.0.fc_low_water: 29220 > dev.em.0.queue0.txd_head: 136 > dev.em.0.queue0.txd_tail: 136 > dev.em.0.queue0.tx_irq: 0 > dev.em.0.queue0.no_desc_avail: 0 > dev.em.0.queue0.rxd_head: 2 > dev.em.0.queue0.rxd_tail: 1 > dev.em.0.queue0.rx_irq: 0 > dev.em.0.mac_stats.excess_**coll: 0 > dev.em.0.mac_stats.single_**coll: 0 > dev.em.0.mac_stats.multiple_**coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_**count: 0 > dev.em.0.mac_stats.symbol_**errors: 0 > dev.em.0.mac_stats.sequence_**errors: 0 > dev.em.0.mac_stats.defer_**count: 0 > dev.em.0.mac_stats.missed_**packets: 0 > dev.em.0.mac_stats.recv_no_**buff: 0 > dev.em.0.mac_stats.recv_**undersize: 0 > dev.em.0.mac_stats.recv_**fragmented: 0 > dev.em.0.mac_stats.recv_**oversize: 0 > dev.em.0.mac_stats.recv_**jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_**errs: 0 > dev.em.0.mac_stats.coll_ext_**errs: 0 > dev.em.0.mac_stats.xon_recvd: 0 > dev.em.0.mac_stats.xon_txd: 0 > dev.em.0.mac_stats.xoff_recvd: 0 > dev.em.0.mac_stats.xoff_txd: 0 > dev.em.0.mac_stats.total_pkts_**recvd: 69774324 > dev.em.0.mac_stats.good_pkts_**recvd: 69774324 > dev.em.0.mac_stats.bcast_pkts_**recvd: 28156 > dev.em.0.mac_stats.mcast_pkts_**recvd: 1758 > dev.em.0.mac_stats.rx_frames_**64: 54177 > dev.em.0.mac_stats.rx_frames_**65_127: 30157358 > dev.em.0.mac_stats.rx_frames_**128_255: 1092948 > dev.em.0.mac_stats.rx_frames_**256_511: 125295 > dev.em.0.mac_stats.rx_frames_**512_1023: 128081 > dev.em.0.mac_stats.rx_frames_**1024_1522: 38216465 > dev.em.0.mac_stats.good_**octets_recvd: 59938624047 > dev.em.0.mac_stats.good_**octets_txd: 106613836902 > dev.em.0.mac_stats.total_pkts_**txd: 96250538 > dev.em.0.mac_stats.good_pkts_**txd: 96250538 > dev.em.0.mac_stats.bcast_pkts_**txd: 2989 > dev.em.0.mac_stats.mcast_pkts_**txd: 0 > dev.em.0.mac_stats.tx_frames_**64: 7551 > dev.em.0.mac_stats.tx_frames_**65_127: 26727682 > dev.em.0.mac_stats.tx_frames_**128_255: 227574 > dev.em.0.mac_stats.tx_frames_**256_511: 167383 > dev.em.0.mac_stats.tx_frames_**512_1023: 302141 > dev.em.0.mac_stats.tx_frames_**1024_1522: 68818207 > dev.em.0.mac_stats.tso_txd: 17244234 > dev.em.0.mac_stats.tso_ctx_**fail: 0 > dev.em.0.interrupts.asserts: 65945396 > dev.em.0.interrupts.rx_pkt_**timer: 8917 > dev.em.0.interrupts.rx_abs_**timer: 0 > dev.em.0.interrupts.tx_pkt_**timer: 1461 > dev.em.0.interrupts.tx_abs_**timer: 1951 > dev.em.0.interrupts.tx_queue_**empty: 0 > dev.em.0.interrupts.tx_queue_**min_thresh: 0 > dev.em.0.interrupts.rx_desc_**min_thresh: 0 > dev.em.0.interrupts.rx_**overrun: 0 > > Interface is RUNNING and INACTIVE > em0: hw tdh = 221, hw tdt = 221 > em0: hw rdh = 467, hw rdt = 466 > em0: Tx Queue Status = 0 > em0: TX descriptors avail = 1024 > em0: Tx Descriptors avail failure = 0 > em0: RX discarded packets = 0 > em0: RX Next to Check = 467 > em0: RX Next to Refresh = 466 > $ > > from 8.2: > > Name Mtu Network Address Ipkts Ierrs Idrop Ibytes > Opkts Oerrs Obytes Coll Drop > em0 1500 00:30:48:2e:99:ba 276150 0 0 42614583 > 285398 0 207023352 0 0 em0 1500 192.147.25.0/192.147.25.65 318062 - - 48331105 285363 - 203026287 > - - em0 1500 fe80::230:48f fe80::230:48ff:fe 0 - - > 0 1 - 96 - - em0 1500 192.147.25.0/192.147.25.45 25071 - - 1782211 0 - 0 > - - em0 1500 192.147.25.0/ 192.147.25.11 38433 - > - 2742827 0 - 0 - - > > > dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=2 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x1079 subvendor=0x15d9 > subdevice=0x117a class=0x020000 > dev.em.0.%parent: pci3 > dev.em.0.nvm: -1 > dev.em.0.rx_int_delay: 0 > dev.em.0.tx_int_delay: 66 > dev.em.0.rx_abs_int_delay: 66 > dev.em.0.tx_abs_int_delay: 66 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 3 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 0 > dev.em.0.tx_desc_fail1: 0 > dev.em.0.tx_desc_fail2: 0 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1089471041 > dev.em.0.rx_control: 32770 > dev.em.0.fc_high_water: 47104 > dev.em.0.fc_low_water: 45604 > dev.em.0.fifo_workaround: 0 > dev.em.0.fifo_reset: 0 > dev.em.0.txd_head: 73 > dev.em.0.txd_tail: 75 > dev.em.0.rxd_head: 130 > dev.em.0.rxd_tail: 129 > dev.em.0.mac_stats.excess_**coll: 0 > dev.em.0.mac_stats.single_**coll: 0 > dev.em.0.mac_stats.multiple_**coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_**count: 0 > dev.em.0.mac_stats.symbol_**errors: 0 > dev.em.0.mac_stats.sequence_**errors: 0 > dev.em.0.mac_stats.defer_**count: 0 > dev.em.0.mac_stats.missed_**packets: 0 > dev.em.0.mac_stats.recv_no_**buff: 0 > dev.em.0.mac_stats.recv_**undersize: 0 > dev.em.0.mac_stats.recv_**fragmented: 0 > dev.em.0.mac_stats.recv_**oversize: 0 > dev.em.0.mac_stats.recv_**jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_**errs: 0 > dev.em.0.mac_stats.coll_ext_**errs: 0 > dev.em.0.mac_stats.xon_recvd: 0 > dev.em.0.mac_stats.xon_txd: 0 > dev.em.0.mac_stats.xoff_recvd: 0 > dev.em.0.mac_stats.xoff_txd: 0 > dev.em.0.mac_stats.total_pkts_**recvd: 276318 > dev.em.0.mac_stats.good_pkts_**recvd: 276318 > dev.em.0.mac_stats.bcast_pkts_**recvd: 8 > dev.em.0.mac_stats.mcast_pkts_**recvd: 0 > dev.em.0.mac_stats.rx_frames_**64: 9012 > dev.em.0.mac_stats.rx_frames_**65_127: 205540 > dev.em.0.mac_stats.rx_frames_**128_255: 44078 > dev.em.0.mac_stats.rx_frames_**256_511: 3166 > dev.em.0.mac_stats.rx_frames_**512_1023: 3934 > dev.em.0.mac_stats.rx_frames_**1024_1522: 10588 > dev.em.0.mac_stats.good_**octets_recvd: 43761022 > dev.em.0.mac_stats.good_**octets_txd: 208238998 > dev.em.0.mac_stats.total_pkts_**txd: 285534 > dev.em.0.mac_stats.good_pkts_**txd: 285534 > dev.em.0.mac_stats.bcast_pkts_**txd: 22 > dev.em.0.mac_stats.mcast_pkts_**txd: 3 > dev.em.0.mac_stats.tx_frames_**64: 6172 > dev.em.0.mac_stats.tx_frames_**65_127: 75983 > dev.em.0.mac_stats.tx_frames_**128_255: 53030 > dev.em.0.mac_stats.tx_frames_**256_511: 23216 > dev.em.0.mac_stats.tx_frames_**512_1023: 1472 > dev.em.0.mac_stats.tx_frames_**1024_1522: 125661 > dev.em.0.mac_stats.tso_txd: 0 > dev.em.0.mac_stats.tso_ctx_**fail: 0 > > $ sudo sysctl dev.em.0.debug=1 > sysctl: unknown oid 'dev.em.0.debug' > $ > > -- > Larry Rosenman http://www.lerctr.org/~ler > Phone: +1 512-248-2683 E-Mail: ler@lerctr.org > US Mail: 430 Valona Loop, Round Rock, TX 78681-3893 > ______________________________**_________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-**stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@**freebsd.org > " >