From owner-freebsd-net@FreeBSD.ORG Fri Jan 2 19:27:12 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 996B916A4CE for ; Fri, 2 Jan 2004 19:27:12 -0800 (PST) Received: from alicia.nttmcl.com (alicia.nttmcl.com [216.69.69.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 81BD843D48 for ; Fri, 2 Jan 2004 19:27:08 -0800 (PST) (envelope-from kelly@nttmcl.com) Received: from alicia.nttmcl.com (localhost [127.0.0.1]) by alicia.nttmcl.com (8.12.9/8.12.5) with ESMTP id i033R8HB045957 for ; Fri, 2 Jan 2004 19:27:08 -0800 (PST) (envelope-from kelly@nttmcl.com) Received: from localhost (kelly@localhost)i033R8MG045954 for ; Fri, 2 Jan 2004 19:27:08 -0800 (PST) (envelope-from kelly@nttmcl.com) X-Authentication-Warning: alicia.nttmcl.com: kelly owned process doing -bs Date: Fri, 2 Jan 2004 19:27:08 -0800 (PST) From: Kelly Yancey To: net@freebsd.org Message-ID: <20040102185125.B45340-100000@alicia.nttmcl.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: 1168 octets payload and bad TCP checksums X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Jan 2004 03:27:12 -0000 We've got Broadcom BCM5701 cards configured for vlan tagging on a FreeBSD 4.7 based router; a vlan switch then terminates the trunked segment and splits it into separate physical subnets. It turns out that hosts on those segments cannot receive TCP packets with precisely 1168 octets of payload (ethernet frame size 1222 octets) as the checksum is always incorrect. We've manually backported all of the bge driver updates from 4-stable, but to no avail. What is particularly odd is that the checksums are always wrong by the same amount: 0xAC48 (the dump below only shows retries of the same packet, but the difference is the same even for other packets). Furthermore, it appears only TCP packets with 1168 octets of data are affected. I cannot easily create an environment without the vlans to determine whether or not tagging is related. Note also, that the IP checksum is correct. Has anyone else experienced similar problems? Does anyone have a clue where to begin to track down the problem? Currently I'm looking at the tcp checksum calculation (tcp_fillheaders), but I don't really see how that could be the culprit as if such a bug existed there, it would affect all interfaces and surely would have been noticed by now. At the same time, I don't see anywhere else offhand the problem could be. Again, if anyone has any advice, I would greatly appreciate it. Thanks, Kelly bge0: flags=8843 mtu 1504 options=3 inet 10.30.3.254 netmask 0xfffffff8 broadcast 10.30.3.255 ether 00:00:5e:00:01:4b media: Ethernet autoselect (100baseTX ) status: active vlan9: flags=8843 mtu 1500 inet 10.30.3.1 netmask 0xfffffff8 broadcast 10.30.3.7 ether 00:00:5e:00:01:4b vlan: 9 parent interface: bge0 vlan10: flags=8843 mtu 1500 inet 10.30.3.9 netmask 0xfffffff8 broadcast 10.30.3.15 ether 00:00:5e:00:01:4b vlan: 10 parent interface: bge0 Extract from tcpdump -vvv taken on host 216.69.90.56 connected to FreeBSD router via vlan10 interface: 11:38:55.665425 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 561:2021(1460) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57881, len 1500) 11:38:55.666782 216.69.68.198.22 > 216.69.90.56.3335: P [tcp sum ok] 2021:2049(28) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57882, len 68) 11:38:55.666839 216.69.90.56.3335 > 216.69.68.198.22: . [tcp sum ok] 432:432(0) ack 2049 win 17520 (DF) (ttl 128, id 57057, len 40) 11:38:55.668899 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57883, len 1208) 11:38:55.920110 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57884, len 1208) 11:38:56.419788 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57885, len 1208) 11:38:56.442824 216.69.224.134 > 216.69.90.56: icmp: echo request (ttl 108, id 24195, len 92) 11:38:57.419622 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57886, len 1208) 11:38:58.098535 216.69.90.56.3337 > 216.69.68.197.53: [udp sum ok] 12575+ PTR? 56.90.69.216.in-addr.arpa. (43) (ttl 128, id 57060, len 71) 11:38:58.098868 216.69.90.56.3337 > 216.69.68.197.53: [udp sum ok] 12576+ PTR? 1.90.69.216.in-addr.arpa. (42) (ttl 128, id 57061, len 70) 11:38:58.102453 216.69.68.197.53 > 216.69.90.56.3337: [udp sum ok] 12575 NXDomain* q: PTR? 56.90.69.216.in-addr.arpa. 0/1/0 ns: 90.69.216.in-addr.arpa. SOA ns.nttmcl.com. hostmaster.nttmcl.com. 2002111000 7200 3600 1209600 432000 (103) (ttl 59, id 43147, len 131) 11:38:58.103689 216.69.68.197.53 > 216.69.90.56.3337: [udp sum ok] 12576 NXDomain* q: PTR? 1.90.69.216.in-addr.arpa. 0/1/0 ns: 90.69.216.in-addr.arpa. SOA ns.nttmcl.com. hostmaster.nttmcl.com. 2002111000 7200 3600 1209600 432000 (102) (ttl 59, id 63562, len 130) 11:38:59.419902 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57887, len 1208) 11:39:03.419776 216.69.68.198.22 > 216.69.90.56.3335: P [bad tcp cksum 1de3!] 2049:3217(1168) ack 432 win 14352 (DF) [tos 0x10] (ttl 59, id 57888, len 1208) 11:39:06.305954 216.69.90.56.3335 > 216.69.68.198.22: P [tcp sum ok] 432:480(48) ack 2049 win 17520 (DF) (ttl 128, id 57062, len 88) 11:39:06.344820 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 3217:3217(0) ack 480 win 14352 (DF) [tos 0x10] (ttl 59, id 57889, len 40) 11:39:07.031807 216.69.90.56.3335 > 216.69.68.198.22: P [tcp sum ok] 480:528(48) ack 2049 win 17520 (DF) (ttl 128, id 57065, len 88) 11:39:07.035322 216.69.68.198.22 > 216.69.90.56.3335: . [tcp sum ok] 3217:3217(0) ack 528 win 14352 (DF) [tos 0x10] (ttl 59, id 57890, len 40)