From owner-freebsd-stable@FreeBSD.ORG Mon Oct 10 23:02:09 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 147A81065674 for ; Mon, 10 Oct 2011 23:02:09 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id ED7E48FC14 for ; Mon, 10 Oct 2011 23:02:08 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta13.emeryville.ca.mail.comcast.net with comcast id j2TC1h0031afHeLADB21GC; Mon, 10 Oct 2011 23:02:01 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta17.emeryville.ca.mail.comcast.net with comcast id jAqg1h00Y1t3BNj8dAqgRE; Mon, 10 Oct 2011 22:50:40 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5CFBD102C1C; Mon, 10 Oct 2011 16:02:07 -0700 (PDT) Date: Mon, 10 Oct 2011 16:02:07 -0700 From: Jeremy Chadwick To: Larry Rosenman Message-ID: <20111010230207.GA85243@icarus.home.lan> References: <201110101147.30558.jhb@freebsd.org> <4E933BBF.6070209@lerctr.org> <36C97D31-5D01-4AC2-8E48-9A8B04B98F91@transsys.com> <4E93606D.8070306@lerctr.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E93606D.8070306@lerctr.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , John Baldwin Subject: Re: rsync corrupted MAC X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2011 23:02:09 -0000 On Mon, Oct 10, 2011 at 04:15:25PM -0500, Larry Rosenman wrote: > On 10/10/2011 3:57 PM, Louis Mamakos wrote: > >On Oct 10, 2011, at 2:38 PM, Larry Rosenman wrote: > > > >>On 10/10/2011 10:47 AM, John Baldwin wrote: > >>>On Sunday, October 09, 2011 5:06:26 pm Larry Rosenman wrote: > >>>>Any ideas on which side or what might be broke here? > >>>> > >>>>ler/MAIL-ARCHIVE/2008/12/INBOX > >>>>Corrupted MAC on input. > >>>>Disconnecting: Packet corrupt > >>>>rsync: connection unexpectedly closed (33845045 bytes received so far) > >>>[receiver] > >>>>rsync error: error in rsync protocol data stream (code 12) at io.c(605) > >>>[receiver=3.0.9] > >>>>rsync: connection unexpectedly closed (1450 bytes received so far) > >>>[generator] > >>>>rsync error: unexplained error (code 255) at io.c(605) [generator=3.0.9] > >>>I've had somewhat similar issues (ssh getting corruption in its data stream) > >>>when a NIC in my netbook was corrupting packet data when it ran at 1G (it > >>>worked fine at 10/100). Pyun eventually fixed the issue by applying enough > >>>workarounds (it was likely a hardware bug in the NIC's chipset). However, it > >>>wasn't easy to debug unfortunately. :( > >>> > >>Any ideas on where to start? > >> > >>from the 8.2 box (tbh.lerctr.org in the script): > >> > >>8.2->PIX->Provider->Internet->Motorola SBG6580 (Time-Warner)->Trendnet TEG-160WS Gig switch->9.0 box (borg.lerctr.org). > >> > >>So, where do I start? > >I'd turn off IP / TCP / UDP checksum offloading on your NIC if it supports it, and see if you are getting network layer checksum errors. If the IP checksum is wrong, then it happened on the last hops between the NIC and memory or across the previous network hop. > > > > > > > Good idea, but, it didn't show ANY errors on EITHER side (both are > em nics). > > Next? > $ ifconfig em0 > em0: flags=8843 metric 0 mtu 1500 > options=2098 > ether 00:30:48:2e:99:ba > inet 192.147.25.65 netmask 0xffffff00 broadcast 192.147.25.255 > inet6 fe80::230:48ff:fe2e:99ba%em0 prefixlen 64 scopeid 0x1 > inet 192.147.25.45 netmask 0xffffff00 broadcast 192.147.25.255 > inet 192.147.25.11 netmask 0xffffff00 broadcast 192.147.25.255 > nd6 options=3 > media: Ethernet autoselect (100baseTX ) > status: active > $ > $ uname -a > FreeBSD thebighonker.lerctr.org 8.2-STABLE FreeBSD 8.2-STABLE #45: > Sat Oct 8 10:57:43 CDT 2011 > root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER > amd64 > $ > > > > $ ifconfig em0 > em0: flags=8843 metric 0 mtu 1500 > options=2088 > ether 00:30:48:8e:9f:f3 > inet 192.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255 > inet6 fe80::230:48ff:fe8e:9ff3%em0 prefixlen 64 scopeid 0x1 > nd6 options=29 > media: Ethernet autoselect (1000baseT ) > status: active > $ uname -a > FreeBSD borg.lerctr.org 9.0-BETA3 FreeBSD 9.0-BETA3 #1: Sun Oct 9 > 10:03:42 CDT 2011 > root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 > $ Can you please provide output from the following commands executed on the machine showing the problem? The above commands show nothing useful, other than the fact that one machine is at 100/full and the other is at 1000/full (I don't know your network setup). Commands: * netstat -inbd -I em0 * sysctl -a dev.em.0 * Issue command "sysctl dev.em.0.debug=1", then type "dmesg" and provide all of the new output you will see at the bottom that pertains to the NIC If you Google this problem, you will find that the majority of the time it's caused by NIC drivers acting oddly. Also, I believe the em(4) driver in 9.x is slightly different than on 8.x, so I'm CC'ing Jack Vogel here. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |