From owner-freebsd-stable@FreeBSD.ORG Thu Feb 14 15:55:32 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 634CBAB9 for ; Thu, 14 Feb 2013 15:55:32 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from qmta15.emeryville.ca.mail.comcast.net (qmta15.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:44:76:96:27:228]) by mx1.freebsd.org (Postfix) with ESMTP id 3B44DC41 for ; Thu, 14 Feb 2013 15:55:32 +0000 (UTC) Received: from omta18.emeryville.ca.mail.comcast.net ([76.96.30.74]) by qmta15.emeryville.ca.mail.comcast.net with comcast id 0EFW1l0051bwxycAFFvY0r; Thu, 14 Feb 2013 15:55:32 +0000 Received: from koitsu.strangled.net ([67.180.84.87]) by omta18.emeryville.ca.mail.comcast.net with comcast id 0FvW1l0131t3BNj8eFvWcd; Thu, 14 Feb 2013 15:55:31 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 90C3973A1C; Thu, 14 Feb 2013 07:55:30 -0800 (PST) Date: Thu, 14 Feb 2013 07:55:30 -0800 From: Jeremy Chadwick To: Doug Hardie Subject: Re: Unusual TCP/IP Packet Size Message-ID: <20130214155530.GA86144@icarus.home.lan> References: <96AE8BD1-79C2-4743-854F-B8386C54E4A1@lafn.org> <511B6B21.5030606@rdtc.ru> <20130213130059.GA57337@icarus.home.lan> <20130214013723.GB2945@michelle.cdnetworks.com> <20130214064521.GA1464@michelle.cdnetworks.com> <3BB4EC29-0FD5-4F5D-9189-51770E2B55D5@lafn.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3BB4EC29-0FD5-4F5D-9189-51770E2B55D5@lafn.org> User-Agent: Mutt/1.5.21 (2010-09-15) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20121106; t=1360857332; bh=U2dQs4gOUs2JIb/nwLKDCFFJBb5T/rOR6ZSxpkzW+Ws=; h=Received:Received:Received:Date:From:To:Subject:Message-ID: MIME-Version:Content-Type; b=mGkx+0HGi1+pxp3IiIluMC5U8e2BUYSqRaWi2KNoVqG3qvN+WybXPRH+hGyHVJ/Fp 9SEhwLiQWoDYhYAotNbwMY1O5/ComtvYHjRMy7/hlusBuwHXxbBqzTtrgBmkKkMTiJ K8kldfs52xL3USXzj6WYsGWstYjuMyUvGkVQG6BxrZNt/2jrQarMSIWfDZdQ3SWOv+ CMrNRZleqPwtP3TYtt/+ttPzawmm7K3rGNwm9RaxUdGyiB9pV5RPISZ62zz5eGS2EV iYXWQVaXLCcV2rouYOzTaUJ3RNwIhIK7ycV/+TXakniJW9YlbZG6qw3E46vL0m6DAZ EZX7ADe506zlg== Cc: pyunyh@gmail.com, freebsd-stable@freebsd.org, Eugene Grosbein , yongari@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Feb 2013 15:55:32 -0000 On Wed, Feb 13, 2013 at 11:54:24PM -0800, Doug Hardie wrote: > > On 13 February 2013, at 22:45, YongHyeon PYUN wrote: > > > On Wed, Feb 13, 2013 at 09:10:36PM -0800, Kevin Oberman wrote: > >> On Wed, Feb 13, 2013 at 5:37 PM, YongHyeon PYUN wrote: > >>> On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote: > >>>> On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: > >>>>> 13.02.2013 17:25, Doug Hardie ??????????: > >>>>>> Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the following interface: > >>>>>> > >>>>>> msk0: flags=8843 metric 0 mtu 1500 > >>>>>> options=c011b > >>>>>> ether 00:11:2f:2a:c7:03 > >>>>>> inet 10.0.1.199 netmask 0xffffff00 broadcast 10.0.1.255 > >>>>>> inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 > >>>>>> nd6 options=29 > >>>>>> media: Ethernet autoselect (100baseTX ) > >>>>>> status: active > >>>>>> > >>>>>> > >>>>>> It sent the following packet: (data content abbreviated) > >>>>>> > >>>>>> 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr 920110183], length 3946 > >>>>>> 0x0000: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.....@.@.*..... > >>>>>> 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc .......J..h..7.. > >>>>>> 0x0020: 8018 0410 3407 0000 0101 080a 17f3 8ff8 ....4...??????. > >>>>>> > >>>>>> > >>>>>> The indicated packet length is 3946 and the load of data shown is that size. The MTU on both interfaces is 1500. The receiving system received 3 packets. There is a router and switch between them. One of them fragmented that packet. This is part of a SSL/TLS exchange and one side or the other is hanging on this and just dropping the connection. I suspect the packet size is the issue. ssldump complains about the packet too and stops monitoring. Could this possibly be related to the hardware checksums? > >>>>> > >>>>> You have TSO enabled on the interface, so large outgoing TCP packet is pretty normal. > >>>>> It will be split by the NIC. Disable TSO with ifconfig if it interferes with your ssldump. > >>>> > >>>> This is not the behaviour I see with em(4) on a 82573E with all defaults > >>>> used (which includes TSO4). Note that Doug is using msk(4). > >>>> > >>>> I can provide packet captures on both ends of a LAN segment using both > >>>> tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that > >>>> show a difference in behaviour compared to what Doug sees. > >>> > >>> This is strange. tcpdump sees a (big) TCP segment right before > >>> controller actually transmits it. So if TSO is active for the TCP > >>> segment, you should see a series of small TCP packets on receiver > >>> side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP > >>> segment with tcpdump on TX path, probably TSO was not used for the > >>> TCP segment. > >>> It's possible for controller to corrupt the TCP segment during > >>> segmentation but Doug's tcpdump looks completely normal to me since > >>> tcpdump sees the segment before TCP segmentation. > >>> > >>>> > >>>> What I see on the FreeBSD side with tcpdump is repeated "bad-len 0" > >>>> messages for payloads which are chunked or segmented as a result of TSO. > >>>> I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I > >>>> only see one "bad-len" entry for all chunks (up until the next ACK or > >>>> PSH+ACK of course). > >>>> > >>> > >>> I vaguely recall that some users reported similar TSO issues on > >>> various drivers. The root cause of the issue was not identified > >>> though. Personally I couldn't reproduce the issue at that time. > >>> It could be a driver or network stack bug. > >> > >> Beware TSO. It can significantly improve throughput on high speed > >> networks, but it really has issues. > >> > >> TSO segments the data and transmits all of them back-to-back with no > >> delay beyond IFG (the 802.3 mandated space between frames) TSO does > >> not understand congestion control. If there is congestion and TSO > >> sends several frames in a row, it is entirely possible that a queue is > >> full or getting close enough to full to start dropping packets and > >> these segmented frames are excellent candidates. > > > > I'm not saying the drawback of TSO. Sometimes segmented packets > > have malformed IP header length under certain circumstances such > > that these packets were dropped on receiver side. > > How do I configure the msk0 interface in rc.conf to disable tso4? I can easily do it with ifconfig, but don't see how to make sure its disabled after a boot. Adjust your ifconfig_msk0 line in rc.conf to contain "-tso", i.e.: ifconfig_msk0="inet 10.0.1.199 netmask 255.255.255.0 -tso" -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |