From owner-freebsd-stable@FreeBSD.ORG Wed Oct 24 18:55:26 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 94B01B37 for ; Wed, 24 Oct 2012 18:55:26 +0000 (UTC) (envelope-from jdc@koitsu.strangled.net) Received: from qmta02.emeryville.ca.mail.comcast.net (qmta02.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:24]) by mx1.freebsd.org (Postfix) with ESMTP id 6EA228FC08 for ; Wed, 24 Oct 2012 18:55:26 +0000 (UTC) Received: from omta18.emeryville.ca.mail.comcast.net ([76.96.30.74]) by qmta02.emeryville.ca.mail.comcast.net with comcast id F3qn1k0021bwxycA26vSRC; Wed, 24 Oct 2012 18:55:26 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta18.emeryville.ca.mail.comcast.net with comcast id F6vR1k0051t3BNj8e6vRiK; Wed, 24 Oct 2012 18:55:26 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 2CFE173A1A; Wed, 24 Oct 2012 11:55:25 -0700 (PDT) Date: Wed, 24 Oct 2012 11:55:25 -0700 From: Jeremy Chadwick To: Harald Schmalzbauer Subject: Re: every 2nd echo-request malformed when ping -s >4067 Message-ID: <20121024185525.GA6426@icarus.home.lan> References: <20121024154017.GA3167@icarus.home.lan> <5088163E.2090506@omnilan.de> <20121024165148.GA4250@icarus.home.lan> <50881EC7.9030400@omnilan.de> <20121024174425.GA4699@icarus.home.lan> <50882D3B.5050704@omnilan.de> <20121024181239.GA5755@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20121024181239.GA5755@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Adrian Chadd , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 18:55:26 -0000 On Wed, Oct 24, 2012 at 11:12:39AM -0700, Jeremy Chadwick wrote: > On Wed, Oct 24, 2012 at 08:02:35PM +0200, Harald Schmalzbauer wrote: > > Please find attached the requested info. > > Thanks, got 'em! I'll reply in a follow-up mail with the decoded > results. As promised, here are the decoded results. Took me longer than I expected since I started going down the road of IP options and then was like, "no, wait a minute, this is ICMP gah!". Opinions are at the bottom. Gosh I hope I didn't botch a copy-paste on this one... 17:58:08.481888 IP 10.5.49.126 > 10.5.49.65: ICMP echo request, id 49423, seq 0, length 4076 0x0000: 4500 1000 1fff 4000 4001 9435 0a05 317e 0x0010: 0a05 3141 0800 a352 c10f 0000 5088 2c30 0x0020: 0007 5a3b {...snip...} 0x45 = bits 7-4: IPv4 protocol = bits 3-0: header length: 20 bytes 0x00 = DSF / RFC 2474 stuff 0x1000 = datagram length: 4096 bytes 0x1fff = fragment id 0x4000 = bits 15-13: %010 = reserved bit (0), DF bit (1), MF bit (0) = bits 12-0: fragment offset: 0 0x40 = TTL: 64 0x01 = protocol: 1 (ICMP) 0x9435 = header checksum 0x0a05317e = source IP 0x0a053141 = destination IP 0x08 = ICMP type: 8 = Echo Request 0x00 = ICMP code: 0 = always zero for ICMP type 8 0xa352 = ICMP header checksum 0xc10f = ICMP identifier 0x0000 = ICMP sequence number 0x5088 = timestamp from ICMP data 0x2c30 = timestamp from ICMP data 0x0007 = timestamp from ICMP data 0x5a3b = timestamp from ICMP data 17:58:09.488461 IP 10.5.49.126 > 10.5.49.65: icmp 0x0000: 4500 1000 1fff 0040 4001 d3f5 0a05 317e 0x0010: 0a05 3141 0800 8998 c10f 0001 5088 2c31 0x0020: 0007 73f3 {...snip...} 0x45 = bits 7-4: IPv4 protocol = bits 3-0: header length: 20 bytes 0x00 = DSF / RFC 2474 stuff 0x1000 = datagram length: 4096 bytes 0x1fff = fragment id 0x0040 = bits 15-13: %000 = reserved bit (0), DF bit (0), MF bit (0) = bits 12-0: fragment offset: 64 0x40 = TTL: 64 0x01 = protocol: 1 (ICMP) 0xd3f5 = header checksum 0x0a05317e = source IP 0x0a053141 = destination IP 0x08 = ICMP type: 8 = Echo Request 0x00 = ICMP code: 0 = always zero for ICMP type 8 0x8998 = ICMP header checksum 0xc10f = ICMP identifier 0x0001 = ICMP sequence number 0x5088 = timestamp from ICMP data 0x2c31 = timestamp from ICMP data 0x0007 = timestamp from ICMP data 0x73f3 = timestamp from ICMP data Summary: I don't see anything anomalous EXCEPT the ordeal regarding the fragment offset going from 0->64 and the DF bit going from 1->0. Possibly this makes tcpdump throw a fit in some way, I'm not sure. If I had to take a guess? I'd say some code somewhere is incorrectly shifting bits (I'm thinking along the lines of a hardware latch counter) or possibly has a byte-ordering problem. I'd love to say "big vs. little endian issue" except I'd expect that to manifest itself constantly time and not just every other packet. But heck if I know what NIC drivers and their IP stack tie-ins are doing under the hood. The good part is that you can reproduce this reliably. P.S. -- Just saw the mail from Barney -- I think he's referring to IP, TCP, and UDP checksums (you can disable that calculation check using tcpdump -K), but that's only in tcpdump. There is also the checksum offloading features on NICs (see ifconfig). I have not the slightest idea if these NIC features could cause this problem / make it manifest in this way. P.P.S. -- Just saw Adrian's comment about zero copy sockets. No idea if it's relevant or not, but disable it anyway because it's going to bite you in the butt come FreeBSD 10.x anyway: http://www.freshbsd.org/commit/freebsd/r241955 http://lists.freebsd.org/pipermail/freebsd-current/2012-October/037354.html -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |