From owner-freebsd-bugs@FreeBSD.ORG Tue Sep 21 10:00:43 2004 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EE2D816A4CE for ; Tue, 21 Sep 2004 10:00:43 +0000 (GMT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id BEC4D43D46 for ; Tue, 21 Sep 2004 10:00:43 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.11/8.12.11) with ESMTP id i8LA0hY0032887 for ; Tue, 21 Sep 2004 10:00:43 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.11/8.12.11/Submit) id i8LA0hSm032882; Tue, 21 Sep 2004 10:00:43 GMT (envelope-from gnats) Resent-Date: Tue, 21 Sep 2004 10:00:43 GMT Resent-Message-Id: <200409211000.i8LA0hSm032882@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Aragon Gouveia Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 08E8616A4CE for ; Tue, 21 Sep 2004 09:52:17 +0000 (GMT) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id D5FA843D5E for ; Tue, 21 Sep 2004 09:52:16 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.12.11/8.12.11) with ESMTP id i8L9qGCf041875 for ; Tue, 21 Sep 2004 09:52:16 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.12.11/8.12.11/Submit) id i8L9qGhu041872; Tue, 21 Sep 2004 09:52:16 GMT (envelope-from nobody) Message-Id: <200409210952.i8L9qGhu041872@www.freebsd.org> Date: Tue, 21 Sep 2004 09:52:16 GMT From: Aragon Gouveia To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Subject: kern/71965: TCP MSS issue in combination with ipfw fwd X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Sep 2004 10:00:44 -0000 >Number: 71965 >Category: kern >Synopsis: TCP MSS issue in combination with ipfw fwd >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Sep 21 10:00:43 GMT 2004 >Closed-Date: >Last-Modified: >Originator: Aragon Gouveia >Release: 5.2.1-RELEASE-p9 >Organization: >Environment: FreeBSD gw.home.geek.sh 5.2.1-RELEASE-p9 FreeBSD 5.2.1-RELEASE-p9 #1: Sun Sep 12 15:11:33 SAST 2004 root@gw.home.geek.sh:/usr/obj/usr/src/sys/GW i386 >Description: Below is a copy of my posts to freebsd-net as requested. ----- Hi, A while ago I setup a vtun tunnel between a FreeBSD 4.10-RELEASE machine and a 5.2.1-RELEASE-p9 machine. Initially everything appeared to work great, but I've just stumbled upon a seriously wierd problem that I can't figure out. I know this is not a support forum for the vtun package, but the problem I'm having is consistently reproducable with another VPN type package - OpenVPN. I'm beginning to think the problem is not related to vtun/OpenVPN and was hoping someone could shed some light on my problem. My setup is as follows. My notebook is running 5.2.1-REALEASE-p9 using userland ppp to establish a PPPoE session over an ADSL bridge. Above that, it runs vtun to establish a UDP tunnel to my VPN server sitting at an ISP. The VPN server is the 4.10-RELEASE machine also running vtun 1.6 configured in server mode. The link is configured for no compression, but encryption enabled. Now the link comes up 100% and passes most data fine. I've set my interface MTU down to 1200 on both sides. I'm also source routing on my notebook with ipfw fwd to make sure traffic flows out the links I want them to flow. This is working 100% too. The problem is this: I'm running Apache 2.0.50 on my notebook. If I request a page from my notebook from an outside machine via the VPN, request responses that exceed the interface MTU are simply not sent. For example, if I request a file sized at 1100 bytes, this is the tcpdump transcript running on my notebook, sniffing the VPN interface: tcpdump: listening on tun1 20:31:01.892060 .3159 > .80: S 4115956342:4115956342(0) win 57344 (DF) [tos 0x10] 20:31:01.892228 .80 > .3159: S 1941194202:1941194202(0) ack 4115956343 win 65535 (DF)20:31:01.974087 .3159 > .80: . ack 1 win 57600 (DF) [tos 0x10] 20:31:08.417478 .3159 > .80: P 1:21(20) ack 1 win 57600 (DF) [tos 0x10] 20:31:08.517285 .80 > .3159: . ack 21 win 33120 (DF) 20:31:10.340468 .3159 > .80: P 21:23(2) ack 1 win 57600 (DF) [tos 0x10] 20:31:10.341371 .80 > .3159: P 1:286(285) ack 23 win 33120 (DF) 20:31:10.341412 .80 > .3159: P 286:1386(1100) ack 23 win 33120 (DF) 20:31:10.342143 .80 > .3159: F 1386:1386(0) ack 23 win 33120 (DF) 20:31:10.568480 .3159 > .80: . ack 286 win 57600 (DF) [tos 0x10] 20:31:10.618594 .3159 > .80: . ack 1387 win 57600 (DF) [tos 0x10] 20:31:10.626417 .3159 > .80: F 23:23(0) ack 1387 win 57600 (DF) [tos 0x10] 20:31:10.626532 .80 > .3159: . ack 24 win 33119 (DF) The two important lines being: 20:31:10.341371 .80 > .3159: P 1:286(285) ack 23 win 33120 (DF) 20:31:10.341412 .80 > .3159: P 286:1386(1100) ack 23 win 33120 (DF) The first of these two is the HTTP response header, and the second the actual requested data (1100 bytes as shown). If I try request a file 1400 bytes large (MTU is 1200): tcpdump: listening on tun1 20:35:06.588068 .3161 > .80: S 1461068997:1461068997(0) win 57344 (DF) [tos 0x10] 20:35:06.588242 .80 > .3161: S 1654337904:1654337904(0) ack 1461068998 win 65535 (DF) 20:35:06.659998 .3161 > .80: . ack 1 win 57600 (DF) [tos 0x10] 20:35:10.490089 .3161 > .80: P 1:21(20) ack 1 win 57600 (DF) [tos 0x10] 20:35:10.589617 .80 > .3161: . ack 21 win 33120 (DF) 20:35:11.506613 .3161 > .80: P 21:23(2) ack 1 win 57600 (DF) [tos 0x10] 20:35:11.507306 .80 > .3161: P 1:286(285) ack 23 win 33120 (DF) 20:35:11.716698 .3161 > .80: . ack 286 win 57600 (DF) [tos 0x10] 20:35:16.619379 .3161 > .80: F 23:23(0) ack 286 win 57600 (DF) [tos 0x10] 20:35:17.815936 .3161 > .80: F 23:23(0) ack 286 win 57600 (DF) [tos 0x10] 20:35:20.017123 .3161 > .80: F 23:23(0) ack 286 win 57600 (DF) [tos 0x10] 20:35:24.227404 .3161 > .80: F 23:23(0) ack 286 win 57600 (DF) [tos 0x10] The two important lines being: 20:35:11.507306 .80 > .3161: P 1:286(285) ack 23 win 33120 (DF) 20:35:11.716698 .3161 > .80: . ack 286 win 57600 (DF) [tos 0x10] The first line of the important ones is the HTTP response header. The second one obviously just the TCP acknowledgement. The expected next few packets that should follow the header never get sent. I'm stumped. As I said, I've tried OpenVPN as well but the behaviour is precisely the same. Does anyone have any ideas? Thanks, Aragon ---- Hi, I've figured out the problem, but I'm not sure how to solve it. Here's what I think is the problem. >From a tcpdump transcript: 09:56:37.652907 .4185 > .80: S 487952620:487952620(0) win 57344 (DF) [tos 0x10] 09:56:37.653076 .80 > .4185: S 4069940133:4069940133(0) ack 487952621 win 65535 (DF) is my notebook running Apache. As can be seen above, it's negotiating an MSS of 1452 with the peer, which it should not be doing. The reason it's doing that is because my default route is via an interface with an MTU of 1492 - the tun interface opened by userland ppp for the PPPoE session over my ADSL bridge. As I said, I'm using ipfw fwd to source route packets from (the vtun tunnel interface address) to the vtun tunnel's remote end-point. But I'm guessing MSS is chosen based on the host's routing table. Which makes perfect sense. So to prove my suspicion I added a route on my notebook as follows: route add -host 196.15.a.y 196.15.a.y being the vtun tunnel's remote end-point. Now the tcpdump transcript looks like this: 10:10:21.227506 .2404 > .80: S 996010957:996010957(0) win 57344 (DF) [tos 0x10] 10:10:21.227717 .80 > .2404: S 2935622965:2935622965(0) ack 996010958 win 65535 (DF) The tunnel's interface MTU was set at 1256 when I did this. So the negotiated MSS is now correct and things are working. But I need to be able to route based on source address and ipfw fwd is the only way I know how to do it. Can anyone think of a clever workaround for this? Is there a way to force the TCP stack to use a set MSS regardless of what the routing table and interface MTU say? Thanks, Aragon ---- >How-To-Repeat: >Fix: Not sure exactly. If the TCP stack can see that ipfw is going to forward packets through an interface different to that specified by the routing table, it should be able to negotiate an appropriate MSS for the relevant session. I think the problem here is that TCP thinks the session will be routed out an interface that its getting from the host's routing table. I don't know exactly how all the internals work, so I'll stop talking now. :) >Release-Note: >Audit-Trail: >Unformatted: