Date: Fri, 30 Mar 2012 15:07:22 -0700 From: Jason Wolfe <nitroboost@gmail.com> To: freebsd-net@freebsd.org Subject: Possible interoperability issue with TCP timestamps between FreeBSD and Linux Message-ID: <CAAAm0r2kRt3nyOSmhpT=uQH129LA6p5DgSy3GShBkQmbMEBxhg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
So I'm seeing an issue that appears to be caused by the strict TCP timestamp adherence in the Linux kernel when there are connections being initiated by both sides of a Linux and FreeBSD pair of servers. What is looks like is FreeBSD is only tracking the timestamp for the remote host for connections it initiates, while the Linux box is tracking it globally for any transactions. As you can see below, this discrepancy seems to be causing Linux to ignore SYN packets when FreeBSD is attempting to connect after what they believe should be the proper timestamp has desyncd. You'll see after 3 SYN failures, FreeBSD drops timestamp support and the connection goes through properly, so this is usually causing about a 10s delay on the handshake. Tail end of a valid/successful transfer initiated by FreeBSD 8.2-RELEASE to Linux. This box in particular is 2.6.32, but we've seen it on other kernels: 18:09:22.271473 IP FreeBSD.56757 > Linux.80: Flags [.], ack 322, win 12559, options [nop,nop,TS val 827470242 ecr 1108508974], length 0 18:09:22.271615 IP FreeBSD.56757 > Linux.80: Flags [F.], seq 247, ack 322, win 12579, options [nop,nop,TS val 827470242 ecr 1108508974], length 0 18:09:22.271888 IP FreeBSD.56757 > Linux.80: Flags [.], ack 323, win 12579, options [nop,nop,TS val 827470242 ecr 1108508974], length 0 18:09:22.980495 IP FreeBSD.56777 > Linux.80: Flags [S], seq 1706839142, win 65535, options [mss 1460,nop,wscale 4,sackOK,TS val 827470951 ecr 0], length 0 18:09:22.980665 IP FreeBSD.56777 > Linux.80: Flags [.], ack 3814836667, win 12579, options [nop,nop,TS val 827470951 ecr 1108509044], length 0 18:09:22.980711 IP FreeBSD.56777 > Linux.80: Flags [P.], ack 1, win 12579, options [nop,nop,TS val 827470951 ecr 1108509044], length 247 18:09:23.001063 IP FreeBSD.56777 > Linux.80: Flags [.], ack 322, win 12559, options [nop,nop,TS val 827470971 ecr 1108509047], length 0 18:09:23.001226 IP FreeBSD.56777 > Linux.80: Flags [F.], seq 247, ack 322, win 12579, options [nop,nop,TS val 827470971 ecr 1108509047], length 0 18:09:23.001541 IP FreeBSD.56777 > Linux.80: Flags [.], ack 323, win 12579, options [nop,nop,TS val 827470972 ecr 1108509047], length 0 Here's a request initiated by Linux to FreeBSD. Note how the timestamp value is a lot higher. 18:09:26.513392 IP FreeBSD.1983 > Linux.24747: Flags [S.], seq 2239886569, ack 3865964318, win 65535, options [mss 1460,nop,wscale 4,sackOK,TS val 1857101612 ecr 1108509398], length 0 18:09:26.513680 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 1, win 12579, options [nop,nop,TS val 1857101612 ecr 1108509398], length 0 18:09:26.513692 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12540, options [nop,nop,TS val 1857101612 ecr 1108509398], length 0 18:09:26.559776 IP FreeBSD.1983 > Linux.24747: Flags [P.], ack 622, win 12579, options [nop,nop,TS val 1857101658 ecr 1108509398], length 285 18:09:26.568579 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509402], length 1448 18:09:26.568584 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509402], length 1448 18:09:26.568587 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509402], length 1448 18:09:26.568591 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509402], length 1448 18:09:26.568843 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 1448 18:09:26.568847 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 1448 18:09:26.568850 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 1448 18:09:26.568854 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 1448 18:09:26.568861 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 1448 18:09:26.568864 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 1448 18:09:26.568867 IP FreeBSD.1983 > Linux.24747: Flags [P.], ack 622, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 954 18:09:26.569222 IP FreeBSD.1983 > Linux.24747: Flags [.], ack 623, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 0 18:09:26.569235 IP FreeBSD.1983 > Linux.24747: Flags [F.], seq 15720, ack 623, win 12579, options [nop,nop,TS val 1857101667 ecr 1108509403], length 0 Then, we make another connection initiated by FreeBSD, but with a sequence number that logically follows from the original outbound connection. This fails, until eventually we automatically give up on timestamps. 18:09:34.631349 IP FreeBSD.57007 > Linux.80: Flags [S], seq 704145119, win 65535, options [mss 1460,nop,wscale 4,sackOK,TS val 827482602 ecr 0], length 0 18:09:37.360671 IP FreeBSD.57056 > Linux.80: Flags [S], seq 2653481758, win 65535, options [mss 1460,nop,wscale 4,sackOK,TS val 827485331 ecr 0], length 0 18:09:37.631176 IP FreeBSD.57007 > Linux.80: Flags [S], seq 704145119, win 65535, options [mss 1460,nop,wscale 4,sackOK,TS val 827485602 ecr 0], length 0 18:09:40.360134 IP FreeBSD.57056 > Linux.80: Flags [S], seq 2653481758, win 65535, options [mss 1460,nop,wscale 4,sackOK,TS val 827488331 ecr 0], length 0 18:09:40.831125 IP FreeBSD.57007 > Linux.80: Flags [S], seq 704145119, win 65535, options [mss 1460,nop,wscale 4,sackOK,TS val 827488802 ecr 0], length 0 18:09:43.560084 IP FreeBSD.57056 > Linux.80: Flags [S], seq 2653481758, win 65535, options [mss 1460,nop,wscale 4,sackOK,TS val 827491531 ecr 0], length 0 18:09:44.031076 IP FreeBSD.57007 > Linux.80: Flags [S], seq 704145119, win 65535, options [mss 1460,sackOK,eol], length 0 18:09:44.031280 IP FreeBSD.57007 > Linux.80: Flags [.], ack 4142946180, win 65535, length 0 We confirmed this further by disabled TCP timestamps on the Linux box via the 'net.ipv4.tcp_timestamps' sysctl, and the problem cleared up. Has any one else seen this? Is there a more graceful work around aside from disabling timestamp support in Linux, or is there a sysctl I'm missing in FreeBSD that causes the timestamp to be global for each peer? Thanks, Jason
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAAm0r2kRt3nyOSmhpT=uQH129LA6p5DgSy3GShBkQmbMEBxhg>