From owner-freebsd-stable@FreeBSD.ORG Tue Oct 11 00:43:08 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CB18106564A; Tue, 11 Oct 2011 00:43:08 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id ACD0F8FC08; Tue, 11 Oct 2011 00:43:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Content-Type:MIME-Version:References:Message-ID:In-Reply-To:Subject:cc:To:Sender:From:Date; bh=v60MnPq1e11crzNTFaQE8KnNirPjd3oJA/g3ucx/vcQ=; b=AtdCmgqykx/m/E1MJQmTWHNSl3HUCl2EW3FPy+Lu6qScbjaNGtzrsBH38NazmFhSV6a2czkZ6QVKyItrbmEXqoqPPS/Ze22Infzw+RGOw9cePEUPjaOriunfH9IfEOoEXyL4VOidmv7p5+MLpPRJYRSPSyMyuyMmaTb4Z6tyorE=; Received: from cpe-72-182-3-73.austin.res.rr.com ([72.182.3.73]:56377 helo=[192.168.200.4]) by thebighonker.lerctr.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1RDQQw-0003Rn-4U; Mon, 10 Oct 2011 19:43:04 -0500 Date: Mon, 10 Oct 2011 19:42:58 -0500 (CDT) From: Larry Rosenman Sender: ler@lrosenman.dyndns.org To: Jeremy Chadwick In-Reply-To: <20111010230207.GA85243@icarus.home.lan> Message-ID: References: <201110101147.30558.jhb@freebsd.org> <4E933BBF.6070209@lerctr.org> <36C97D31-5D01-4AC2-8E48-9A8B04B98F91@transsys.com> <4E93606D.8070306@lerctr.org> <20111010230207.GA85243@icarus.home.lan> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: -2.7 (--) X-LERCTR-Spam-Score: -2.7 (--) X-Spam-Report: SpamScore (-2.7/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, TW_TD=0.077, TW_XF=0.077 X-LERCTR-Spam-Report: SpamScore (-2.7/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, TW_TD=0.077, TW_XF=0.077 Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , John Baldwin Subject: Re: rsync corrupted MAC X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Oct 2011 00:43:08 -0000 On Mon, 10 Oct 2011, Jeremy Chadwick wrote: > On Mon, Oct 10, 2011 at 04:15:25PM -0500, Larry Rosenman wrote: >> On 10/10/2011 3:57 PM, Louis Mamakos wrote: >>> On Oct 10, 2011, at 2:38 PM, Larry Rosenman wrote: >>> >>>> On 10/10/2011 10:47 AM, John Baldwin wrote: >>>>> On Sunday, October 09, 2011 5:06:26 pm Larry Rosenman wrote: >>>>>> Any ideas on which side or what might be broke here? >>>>>> >>>>>> ler/MAIL-ARCHIVE/2008/12/INBOX >>>>>> Corrupted MAC on input. >>>>>> Disconnecting: Packet corrupt >>>>>> rsync: connection unexpectedly closed (33845045 bytes received so far) >>>>> [receiver] >>>>>> rsync error: error in rsync protocol data stream (code 12) at io.c(605) >>>>> [receiver=3.0.9] >>>>>> rsync: connection unexpectedly closed (1450 bytes received so far) >>>>> [generator] >>>>>> rsync error: unexplained error (code 255) at io.c(605) [generator=3.0.9] >>>>> I've had somewhat similar issues (ssh getting corruption in its data stream) >>>>> when a NIC in my netbook was corrupting packet data when it ran at 1G (it >>>>> worked fine at 10/100). Pyun eventually fixed the issue by applying enough >>>>> workarounds (it was likely a hardware bug in the NIC's chipset). However, it >>>>> wasn't easy to debug unfortunately. :( >>>>> >>>> Any ideas on where to start? >>>> >>>> from the 8.2 box (tbh.lerctr.org in the script): >>>> >>>> 8.2->PIX->Provider->Internet->Motorola SBG6580 (Time-Warner)->Trendnet TEG-160WS Gig switch->9.0 box (borg.lerctr.org). >>>> >>>> So, where do I start? >>> I'd turn off IP / TCP / UDP checksum offloading on your NIC if it supports it, and see if you are getting network layer checksum errors. If the IP checksum is wrong, then it happened on the last hops between the NIC and memory or across the previous network hop. >>> >>> >>> >> Good idea, but, it didn't show ANY errors on EITHER side (both are >> em nics). >> >> Next? >> $ ifconfig em0 >> em0: flags=8843 metric 0 mtu 1500 >> options=2098 >> ether 00:30:48:2e:99:ba >> inet 192.147.25.65 netmask 0xffffff00 broadcast 192.147.25.255 >> inet6 fe80::230:48ff:fe2e:99ba%em0 prefixlen 64 scopeid 0x1 >> inet 192.147.25.45 netmask 0xffffff00 broadcast 192.147.25.255 >> inet 192.147.25.11 netmask 0xffffff00 broadcast 192.147.25.255 >> nd6 options=3 >> media: Ethernet autoselect (100baseTX ) >> status: active >> $ >> $ uname -a >> FreeBSD thebighonker.lerctr.org 8.2-STABLE FreeBSD 8.2-STABLE #45: >> Sat Oct 8 10:57:43 CDT 2011 >> root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER >> amd64 >> $ >> >> >> >> $ ifconfig em0 >> em0: flags=8843 metric 0 mtu 1500 >> options=2088 >> ether 00:30:48:8e:9f:f3 >> inet 192.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255 >> inet6 fe80::230:48ff:fe8e:9ff3%em0 prefixlen 64 scopeid 0x1 >> nd6 options=29 >> media: Ethernet autoselect (1000baseT ) >> status: active >> $ uname -a >> FreeBSD borg.lerctr.org 9.0-BETA3 FreeBSD 9.0-BETA3 #1: Sun Oct 9 >> 10:03:42 CDT 2011 >> root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 >> $ > > Can you please provide output from the following commands executed on > the machine showing the problem? The above commands show nothing > useful, other than the fact that one machine is at 100/full and the > other is at 1000/full (I don't know your network setup). Commands: > > * netstat -inbd -I em0 > * sysctl -a dev.em.0 > * Issue command "sysctl dev.em.0.debug=1", then type "dmesg" and > provide all of the new output you will see at the bottom that > pertains to the NIC > > If you Google this problem, you will find that the majority of the time > it's caused by NIC drivers acting oddly. > > Also, I believe the em(4) driver in 9.x is slightly different than on > 8.x, so I'm CC'ing Jack Vogel here. > > from 9.0: Name Mtu Network Address Ipkts Ierrs Idrop Ibytes Opkts Oerrs Obytes Coll Drop em0 1500 00:30:48:8e:9f:f3 69776975 0 0 59660392277 52592789 0 104743924118 0 0 em0 1500 192.168.200.0 192.168.200.4 69759773 - - 58681934612 96397272 - 104003761109 - - em0 1500 fe80::230:48f fe80::230:48ff:fe 0 - - 0 3 - 248 - - dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 dev.em.0.%driver: em dev.em.0.%location: slot=0 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x1096 subvendor=0x15d9 subdevice=0x0000 class=0x020000 dev.em.0.%parent: pci6 dev.em.0.nvm: -1 dev.em.0.debug: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.eee_control: 0 dev.em.0.link_irq: 0 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 21755 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1851969 dev.em.0.rx_control: 67141634 dev.em.0.fc_high_water: 30720 dev.em.0.fc_low_water: 29220 dev.em.0.queue0.txd_head: 136 dev.em.0.queue0.txd_tail: 136 dev.em.0.queue0.tx_irq: 0 dev.em.0.queue0.no_desc_avail: 0 dev.em.0.queue0.rxd_head: 2 dev.em.0.queue0.rxd_tail: 1 dev.em.0.queue0.rx_irq: 0 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd: 69774324 dev.em.0.mac_stats.good_pkts_recvd: 69774324 dev.em.0.mac_stats.bcast_pkts_recvd: 28156 dev.em.0.mac_stats.mcast_pkts_recvd: 1758 dev.em.0.mac_stats.rx_frames_64: 54177 dev.em.0.mac_stats.rx_frames_65_127: 30157358 dev.em.0.mac_stats.rx_frames_128_255: 1092948 dev.em.0.mac_stats.rx_frames_256_511: 125295 dev.em.0.mac_stats.rx_frames_512_1023: 128081 dev.em.0.mac_stats.rx_frames_1024_1522: 38216465 dev.em.0.mac_stats.good_octets_recvd: 59938624047 dev.em.0.mac_stats.good_octets_txd: 106613836902 dev.em.0.mac_stats.total_pkts_txd: 96250538 dev.em.0.mac_stats.good_pkts_txd: 96250538 dev.em.0.mac_stats.bcast_pkts_txd: 2989 dev.em.0.mac_stats.mcast_pkts_txd: 0 dev.em.0.mac_stats.tx_frames_64: 7551 dev.em.0.mac_stats.tx_frames_65_127: 26727682 dev.em.0.mac_stats.tx_frames_128_255: 227574 dev.em.0.mac_stats.tx_frames_256_511: 167383 dev.em.0.mac_stats.tx_frames_512_1023: 302141 dev.em.0.mac_stats.tx_frames_1024_1522: 68818207 dev.em.0.mac_stats.tso_txd: 17244234 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.interrupts.asserts: 65945396 dev.em.0.interrupts.rx_pkt_timer: 8917 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.tx_pkt_timer: 1461 dev.em.0.interrupts.tx_abs_timer: 1951 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_queue_min_thresh: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.rx_overrun: 0 Interface is RUNNING and INACTIVE em0: hw tdh = 221, hw tdt = 221 em0: hw rdh = 467, hw rdt = 466 em0: Tx Queue Status = 0 em0: TX descriptors avail = 1024 em0: Tx Descriptors avail failure = 0 em0: RX discarded packets = 0 em0: RX Next to Check = 467 em0: RX Next to Refresh = 466 $ from 8.2: Name Mtu Network Address Ipkts Ierrs Idrop Ibytes Opkts Oerrs Obytes Coll Drop em0 1500 00:30:48:2e:99:ba 276150 0 0 42614583 285398 0 207023352 0 0 em0 1500 192.147.25.0/ 192.147.25.65 318062 - - 48331105 285363 - 203026287 - - em0 1500 fe80::230:48f fe80::230:48ff:fe 0 - - 0 1 - 96 - - em0 1500 192.147.25.0/ 192.147.25.45 25071 - - 1782211 0 - 0 - - em0 1500 192.147.25.0/ 192.147.25.11 38433 - - 2742827 0 - 0 - - dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.3 dev.em.0.%driver: em dev.em.0.%location: slot=2 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x1079 subvendor=0x15d9 subdevice=0x117a class=0x020000 dev.em.0.%parent: pci3 dev.em.0.nvm: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.tx_desc_fail1: 0 dev.em.0.tx_desc_fail2: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1089471041 dev.em.0.rx_control: 32770 dev.em.0.fc_high_water: 47104 dev.em.0.fc_low_water: 45604 dev.em.0.fifo_workaround: 0 dev.em.0.fifo_reset: 0 dev.em.0.txd_head: 73 dev.em.0.txd_tail: 75 dev.em.0.rxd_head: 130 dev.em.0.rxd_tail: 129 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd: 276318 dev.em.0.mac_stats.good_pkts_recvd: 276318 dev.em.0.mac_stats.bcast_pkts_recvd: 8 dev.em.0.mac_stats.mcast_pkts_recvd: 0 dev.em.0.mac_stats.rx_frames_64: 9012 dev.em.0.mac_stats.rx_frames_65_127: 205540 dev.em.0.mac_stats.rx_frames_128_255: 44078 dev.em.0.mac_stats.rx_frames_256_511: 3166 dev.em.0.mac_stats.rx_frames_512_1023: 3934 dev.em.0.mac_stats.rx_frames_1024_1522: 10588 dev.em.0.mac_stats.good_octets_recvd: 43761022 dev.em.0.mac_stats.good_octets_txd: 208238998 dev.em.0.mac_stats.total_pkts_txd: 285534 dev.em.0.mac_stats.good_pkts_txd: 285534 dev.em.0.mac_stats.bcast_pkts_txd: 22 dev.em.0.mac_stats.mcast_pkts_txd: 3 dev.em.0.mac_stats.tx_frames_64: 6172 dev.em.0.mac_stats.tx_frames_65_127: 75983 dev.em.0.mac_stats.tx_frames_128_255: 53030 dev.em.0.mac_stats.tx_frames_256_511: 23216 dev.em.0.mac_stats.tx_frames_512_1023: 1472 dev.em.0.mac_stats.tx_frames_1024_1522: 125661 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0 $ sudo sysctl dev.em.0.debug=1 sysctl: unknown oid 'dev.em.0.debug' $ -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 512-248-2683 E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893