Date: Thu, 14 Feb 2013 12:43:22 -0800 From: Jeremy Chadwick <jdc@koitsu.org> To: YongHyeon PYUN <pyunyh@gmail.com> Cc: freebsd-stable@freebsd.org, Eugene Grosbein <egrosbein@rdtc.ru>, yongari@freebsd.org Subject: Re: Unusual TCP/IP Packet Size Message-ID: <20130214204322.GA90281@icarus.home.lan> In-Reply-To: <20130214013723.GB2945@michelle.cdnetworks.com> References: <96AE8BD1-79C2-4743-854F-B8386C54E4A1@lafn.org> <511B6B21.5030606@rdtc.ru> <20130213130059.GA57337@icarus.home.lan> <20130214013723.GB2945@michelle.cdnetworks.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Feb 14, 2013 at 10:37:23AM +0900, YongHyeon PYUN wrote:
> On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote:
> > On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote:
> > > 13.02.2013 17:25, Doug Hardie ??????????:
> > > > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has the following interface:
> > > >
> > > > msk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > > > options=c011b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE>
> > > > ether 00:11:2f:2a:c7:03
> > > > inet 10.0.1.199 netmask 0xffffff00 broadcast 10.0.1.255
> > > > inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1
> > > > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > > > media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)
> > > > status: active
> > > >
> > > >
> > > > It sent the following packet: (data content abbreviated)
> > > >
> > > > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr 920110183], length 3946
> > > > 0x0000: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.....@.@.*.....
> > > > 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc .......J..h..7..
> > > > 0x0020: 8018 0410 3407 0000 0101 080a 17f3 8ff8 ....4...??????.
> > > >
> > > >
> > > > The indicated packet length is 3946 and the load of data shown is that size. The MTU on both interfaces is 1500. The receiving system received 3 packets. There is a router and switch between them. One of them fragmented that packet. This is part of a SSL/TLS exchange and one side or the other is hanging on this and just dropping the connection. I suspect the packet size is the issue. ssldump complains about the packet too and stops monitoring. Could this possibly be related to the hardware checksums?
> > >
> > > You have TSO enabled on the interface, so large outgoing TCP packet is pretty normal.
> > > It will be split by the NIC. Disable TSO with ifconfig if it interferes with your ssldump.
> >
> > This is not the behaviour I see with em(4) on a 82573E with all defaults
> > used (which includes TSO4). Note that Doug is using msk(4).
> >
> > I can provide packet captures on both ends of a LAN segment using both
> > tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that
> > show a difference in behaviour compared to what Doug sees.
>
> This is strange. tcpdump sees a (big) TCP segment right before
> controller actually transmits it. So if TSO is active for the TCP
> segment, you should see a series of small TCP packets on receiver
> side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP
> segment with tcpdump on TX path, probably TSO was not used for the
> TCP segment.
> It's possible for controller to corrupt the TCP segment during
> segmentation but Doug's tcpdump looks completely normal to me since
> tcpdump sees the segment before TCP segmentation.
Let me explain what I'm referring to.
In the below tcpdump on the server, you'll find 5 "bad-len 0" messages.
These correlate directly with TCP payloads that exceed MTU -- this is
verified by comparing to the number of lines labelled "TCP segment of
reassembled PDU" **that exceed MTU (>1500)** in Wireshark (when
analysing the same server capture).
Thus, the "bad-len 0" messages in tcpdump are indicators of TSO being
used.
Doug's capture with msk(4) + TSO shows a TCP length that exceeds MTU,
yet my capture with em(4) + TSO shows "bad-len 0".
Wireshark seems to decode server capture correctly, so I'm inclined to
think this is a libpcap/tcpdump quirk/bug of sorts. I just find it very
strange that NIC difference manifests itself like this (and in some
regards, concerns me).
I'm happy to provide the pcap files from both server and client if
someone wants to do the analysis, as well as a "verbose" output from
Wireshark (vs. just the summary lines) if all packet fields are desired.
Server = 192.168.1.51 (FreeBSD, Intel 82573E, em(4), with TSO)
Client = 192.168.1.50 (Windows XP SP3)
Server capture (tcpdump -p -i em0 -n -s 0 -w server.pcap):
12:14:54.512542 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [S], seq 375412185, win 65535, options [mss 1460,nop,wscale 3,nop,nop,sackOK], length 0
12:14:54.512576 IP 192.168.1.51.80 > 192.168.1.50.1300: Flags [S.], seq 339580135, ack 375412186, win 65535, options [mss 1460,nop,wscale 6,sackOK,eol], length 0
12:14:54.512659 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 1, win 64240, length 0
12:14:54.512784 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [P.], seq 1:330, ack 1, win 64240, length 329
12:14:54.515194 IP 192.168.1.51.80 > 192.168.1.50.1300: Flags [P.], seq 1:279, ack 330, win 1026, length 278
12:14:54.515230 IP bad-len 0
12:14:54.515410 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 1739, win 64240, length 0
12:14:54.515423 IP bad-len 0
12:14:54.515429 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 4382, win 64240, length 0
12:14:54.515439 IP bad-len 0
12:14:54.515657 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 7858, win 64240, length 0
12:14:54.515667 IP bad-len 0
12:14:54.515674 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 10778, win 64240, length 0
12:14:54.515682 IP bad-len 0
12:14:54.515688 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 13144, win 64240, length 0
12:14:54.515907 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 16064, win 64240, length 0
12:14:54.515913 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 19540, win 64240, length 0
12:14:54.515918 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 22460, win 64240, length 0
12:14:54.515923 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 24354, win 64240, length 0
12:14:59.516217 IP 192.168.1.51.80 > 192.168.1.50.1300: Flags [F.], seq 24354, ack 330, win 1026, length 0
12:14:59.516408 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [.], ack 24355, win 64240, length 0
12:14:59.516425 IP 192.168.1.50.1300 > 192.168.1.51.80: Flags [F.], seq 330, ack 24355, win 64240, length 0
12:14:59.516454 IP 192.168.1.51.80 > 192.168.1.50.1300: Flags [.], ack 331, win 1026, length 0
Server capture (server.pcap) analysed by Wireshark:
No. Time Source Destination Protocol Length Info
1 12:14:54.512542 192.168.1.50 192.168.1.51 TCP 66 1300 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=8 SACK_PERM=1
2 12:14:54.512576 192.168.1.51 192.168.1.50 TCP 66 80 > 1300 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=64 SACK_PERM=1
3 12:14:54.512659 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=1 Ack=1 Win=513920 Len=0
4 12:14:54.512784 192.168.1.50 192.168.1.51 HTTP 383 GET /wtf.jpg HTTP/1.1
5 12:14:54.515194 192.168.1.51 192.168.1.50 TCP 332 [TCP segment of a reassembled PDU]
6 12:14:54.515230 192.168.1.51 192.168.1.50 TCP 4157 [TCP segment of a reassembled PDU]
7 12:14:54.515410 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=1739 Win=513920 Len=0
8 12:14:54.515423 192.168.1.51 192.168.1.50 TCP 3530 [TCP segment of a reassembled PDU]
9 12:14:54.515429 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=4382 Win=513920 Len=0
10 12:14:54.515439 192.168.1.51 192.168.1.50 TCP 5340 [TCP segment of a reassembled PDU]
11 12:14:54.515657 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=7858 Win=513920 Len=0
12 12:14:54.515667 192.168.1.51 192.168.1.50 TCP 6450 [TCP segment of a reassembled PDU]
13 12:14:54.515674 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=10778 Win=513920 Len=0
14 12:14:54.515682 192.168.1.51 192.168.1.50 HTTP 4868 HTTP/1.1 200 OK (JPEG JFIF image)
15 12:14:54.515688 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=13144 Win=513920 Len=0
16 12:14:54.515907 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=16064 Win=513920 Len=0
17 12:14:54.515913 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=19540 Win=513920 Len=0
18 12:14:54.515918 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=22460 Win=513920 Len=0
19 12:14:54.515923 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=24354 Win=513920 Len=0
20 12:14:59.516217 192.168.1.51 192.168.1.50 TCP 54 80 > 1300 [FIN, ACK] Seq=24354 Ack=330 Win=65664 Len=0
21 12:14:59.516408 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [ACK] Seq=330 Ack=24355 Win=513920 Len=0
22 12:14:59.516425 192.168.1.50 192.168.1.51 TCP 60 1300 > 80 [FIN, ACK] Seq=330 Ack=24355 Win=513920 Len=0
23 12:14:59.516454 192.168.1.51 192.168.1.50 TCP 54 80 > 1300 [ACK] Seq=24355 Ack=331 Win=65664 Len=0
Client capture:
No. Time Source Destination Protocol Length Info
1 12:14:54.555808000 192.168.1.50 192.168.1.51 TCP 66 1300 > 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=8 SACK_PERM=1
2 12:14:54.555978000 192.168.1.51 192.168.1.50 TCP 66 80 > 1300 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=64 SACK_PERM=1
3 12:14:54.555988000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=1 Ack=1 Win=513920 Len=0
4 12:14:54.556057000 192.168.1.50 192.168.1.51 HTTP 383 GET /wtf.jpg HTTP/1.1
5 12:14:54.558602000 192.168.1.51 192.168.1.50 TCP 332 [TCP segment of a reassembled PDU]
6 12:14:54.558674000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
7 12:14:54.558684000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=1739 Win=513920 Len=0
8 12:14:54.558703000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
9 12:14:54.558708000 192.168.1.51 192.168.1.50 TCP 1237 [TCP segment of a reassembled PDU]
10 12:14:54.558715000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=4382 Win=513920 Len=0
11 12:14:54.558869000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
12 12:14:54.558883000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
13 12:14:54.558891000 192.168.1.51 192.168.1.50 TCP 610 [TCP segment of a reassembled PDU]
14 12:14:54.558898000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=7858 Win=513920 Len=0
15 12:14:54.558910000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
16 12:14:54.558925000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
17 12:14:54.558931000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=10778 Win=513920 Len=0
18 12:14:54.558947000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
19 12:14:54.558950000 192.168.1.51 192.168.1.50 TCP 960 [TCP segment of a reassembled PDU]
20 12:14:54.558956000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=13144 Win=513920 Len=0
21 12:14:54.559116000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
22 12:14:54.559132000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
23 12:14:54.559140000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=16064 Win=513920 Len=0
24 12:14:54.559157000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
25 12:14:54.559161000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
26 12:14:54.559165000 192.168.1.51 192.168.1.50 TCP 610 [TCP segment of a reassembled PDU]
27 12:14:54.559171000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=19540 Win=513920 Len=0
28 12:14:54.559189000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
29 12:14:54.559193000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
30 12:14:54.559199000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=22460 Win=513920 Len=0
31 12:14:54.559214000 192.168.1.51 192.168.1.50 TCP 1514 [TCP segment of a reassembled PDU]
32 12:14:54.559218000 192.168.1.51 192.168.1.50 HTTP 488 HTTP/1.1 200 OK (JPEG JFIF image)
33 12:14:54.559229000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=24354 Win=513920 Len=0
34 12:14:59.559591000 192.168.1.51 192.168.1.50 TCP 60 80 > 1300 [FIN, ACK] Seq=24354 Ack=330 Win=65664 Len=0
35 12:14:59.559610000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [ACK] Seq=330 Ack=24355 Win=513920 Len=0
36 12:14:59.559668000 192.168.1.50 192.168.1.51 TCP 54 1300 > 80 [FIN, ACK] Seq=330 Ack=24355 Win=513920 Len=0
37 12:14:59.559828000 192.168.1.51 192.168.1.50 TCP 60 80 > 1300 [ACK] Seq=24355 Ack=331 Win=65664 Len=0
--
| Jeremy Chadwick jdc@koitsu.org |
| UNIX Systems Administrator http://jdc.koitsu.org/ |
| Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130214204322.GA90281>