From owner-freebsd-net@FreeBSD.ORG Sat Oct 22 20:35:13 2005 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0058516A41F for ; Sat, 22 Oct 2005 20:35:12 +0000 (GMT) (envelope-from mgrooms@shrew.net) Received: from shrew.net (shrew.net [200.46.204.197]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3886843D46 for ; Sat, 22 Oct 2005 20:35:12 +0000 (GMT) (envelope-from mgrooms@shrew.net) Received: from hole.shrew.net (66-90-165-114.dyn.grandenetworks.net [66.90.165.114]) by shrew.net (Postfix) with ESMTP id F15DF981CB1; Sat, 22 Oct 2005 18:33:31 +0000 (GMT) Received: from [10.22.200.21] ([10.22.200.21]) by hole.shrew.net (8.13.4/8.13.4) with ESMTP id j9MIXSqb077017; Sat, 22 Oct 2005 13:33:30 -0500 (CDT) (envelope-from mgrooms@shrew.net) Message-ID: <435A85F7.3000909@shrew.net> Date: Sat, 22 Oct 2005 13:33:27 -0500 From: Matthew Grooms User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: volker@vwsoft.com Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0 (hole.shrew.net [66.90.165.114]); Sat, 22 Oct 2005 13:33:30 -0500 (CDT) X-Spam-Status: No, score=-2.8 required=5.0 tests=ALL_TRUSTED,AWL autolearn=ham version=3.0.4 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on hole.shrew.net Cc: freebsd-net@freebsd.org Subject: IPSec tcp session stalling ( me too ) ... X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Oct 2005 20:35:13 -0000 Volker, I have noticed the same problem. In my case, it only seems to happen when the traffic is being forwarded across interfaces and pf or ipfw is enabled. I use purely IPSEC so I would agree that GRE isn't the problem. This behavior is 100% reproducible for me. If traffic is forwarded from the host providing the ESP protection or if the firewall package is disabled, the problem goes away. Just some data points. I don't recall seeing this ever happen on 4.x + ipfw. I experienced this on early 5.x + ipfw, late 5.x + pf and 6.x + pf. I believe the ipfw versions I tested were prior to the pfil hooks conversion. For example ... NODE 1 sftp client NODE 2 sftp server IPSEC policy requires ESP protection from NODE 1 or VPN A to NODE 2 NODE 1 ------ VPN A ===== VPN B ----- NODE 2 1) NODE 1 <-> NODE 2 sftp via IPSEC pf enabled, traffic stalls 2) NODE 1 <-> NODE 2 sftp via IPSEC pf disabled, no problems 3) VPN A <-> NODE 2 sftp via IPSEC pf enabled, no problems NOTE : TCP protocol is irrelevant. Haven't tried UDP. tcpdump from VPN A internal interface in the (1) case ... tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on xl1, link-type EN10MB (Ethernet), capture size 96 bytes 12:47:02.673536 IP 10.22.200.21.1223 > 10.20.10.141.ssh: S 929298208:929298208(0) win 16384 12:47:02.714815 IP 10.20.10.141.ssh > 10.22.200.21.1223: S 4112220692:4112220692(0) ack 929298209 win 5840 12:47:02.715005 IP 10.22.200.21.1223 > 10.20.10.141.ssh: . ack 1 win 17520 12:47:04.390884 IP 10.20.10.141.ssh > 10.22.200.21.nerv: P 4052971969:4052972505(536) ack 992811526 win 6432 12:47:04.391082 IP 10.22.200.21.nerv > 10.20.10.141.ssh: . ack 536 win 16319 12:47:06.312301 IP 10.20.10.141.ssh > 10.22.200.21.1223: S 4112220692:4112220692(0) ack 929298209 win 5840 12:47:06.312418 IP 10.22.200.21.1223 > 10.20.10.141.ssh: . ack 1 win 17520 12:47:12.311382 IP 10.20.10.141.ssh > 10.22.200.21.1223: S 4112220692:4112220692(0) ack 929298209 win 5840 12:47:12.311492 IP 10.22.200.21.1223 > 10.20.10.141.ssh: . ack 1 win 17520 12:47:12.349209 IP 10.20.10.141.ssh > 10.22.200.21.1223: P 1:26(25) ack 1 win 5840 12:47:12.349463 IP 10.22.200.21.1223 > 10.20.10.141.ssh: P 1:38(37) ack 26 win 17495 12:47:15.342481 IP 10.20.10.141.ssh > 10.22.200.21.1223: P 1:26(25) ack 1 win 5840 12:47:15.342592 IP 10.22.200.21.1223 > 10.20.10.141.ssh: . ack 26 win 17495 12:47:15.530728 IP 10.22.200.21.1223 > 10.20.10.141.ssh: P 1:446(445) ack 26 win 17495 12:47:22.093189 IP 10.22.200.21.1223 > 10.20.10.141.ssh: P 1:446(445) ack 26 win 17495 12:47:22.140791 IP 10.20.10.141.ssh > 10.22.200.21.1223: . ack 446 win 6432 12:47:22.141931 IP 10.20.10.141.ssh > 10.22.200.21.1223: P 26:666(640) ack 446 win 6432 12:47:22.142232 IP 10.22.200.21.1223 > 10.20.10.141.ssh: P 446:462(16) ack 666 win 16855 12:47:28.136807 IP 10.20.10.141.ssh > 10.22.200.21.1223: P 26:666(640) ack 446 win 6432 12:47:28.137018 IP 10.22.200.21.1223 > 10.20.10.141.ssh: . ack 666 win 16855 12:47:35.218016 IP 10.22.200.21.1223 > 10.20.10.141.ssh: P 446:462(16) ack 666 win 16855 tcpdump from VPN A internal interface in the (2) case ... root@hole# tcpdump -i xl1 src or dst 10.20.10.141 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on xl1, link-type EN10MB (Ethernet), capture size 96 bytes 12:58:36.411839 IP 10.22.200.21.1228 > 10.20.10.141.ssh: S 3459356125:3459356125(0) win 16384 12:58:36.452844 IP 10.20.10.141.ssh > 10.22.200.21.1228: S 548348284:548348284(0) ack 3459356126 win 5840 12:58:36.452954 IP 10.22.200.21.1228 > 10.20.10.141.ssh: . ack 1 win 17520 12:58:40.048592 IP 10.20.10.141.ssh > 10.22.200.21.1228: S 548348284:548348284(0) ack 3459356126 win 5840 12:58:40.048693 IP 10.22.200.21.1228 > 10.20.10.141.ssh: . ack 1 win 17520 12:58:40.086207 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 1:26(25) ack 1 win 5840 12:58:40.086452 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 1:38(37) ack 26 win 17495 12:58:40.120738 IP 10.20.10.141.ssh > 10.22.200.21.1228: . ack 38 win 5840 12:58:40.120915 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 38:446(408) ack 26 win 17495 12:58:40.124873 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 26:666(640) ack 38 win 5840 12:58:40.199077 IP 10.20.10.141.ssh > 10.22.200.21.1228: . ack 446 win 6432 12:58:40.199198 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 446:462(16) ack 666 win 16855 12:58:40.237038 IP 10.20.10.141.ssh > 10.22.200.21.1228: . ack 462 win 6432 12:58:40.245737 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 666:1202(536) ack 462 win 6432 12:58:40.317230 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 462:990(528) ack 1202 win 16319 12:58:40.390865 IP 10.20.10.141.ssh > 10.22.200.21.1228: . ack 990 win 7504 12:58:40.544484 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 1202:2242(1040) ack 990 win 7504 12:58:40.594600 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 990:1006(16) ack 2242 win 17520 12:58:40.628544 IP 10.20.10.141.ssh > 10.22.200.21.1228: . ack 1006 win 7504 12:58:40.628660 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 1006:1054(48) ack 2242 win 17520 12:58:40.663963 IP 10.20.10.141.ssh > 10.22.200.21.1228: . ack 1054 win 7504 12:58:40.664821 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 2242:2290(48) ack 1054 win 7504 12:58:40.665010 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 1054:1118(64) ack 2290 win 17472 12:58:40.723227 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 2290:2370(80) ack 1118 win 7504 12:58:40.723421 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 1118:1198(80) ack 2370 win 17392 12:58:40.783036 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 2370:2402(32) ack 1198 win 7504 12:58:40.783226 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 1198:1262(64) ack 2402 win 17360 12:58:40.817686 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 2402:2450(48) ack 1262 win 7504 12:58:40.817926 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 1262:1342(80) ack 2450 win 17312 12:58:40.856225 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 2450:2482(32) ack 1342 win 7504 12:58:40.856392 IP 10.22.200.21.1228 > 10.20.10.141.ssh: P 1342:1390(48) ack 2482 win 17280 12:58:40.903573 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 2482:2562(80) ack 1390 win 7504 12:58:40.903921 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 2562:2690(128) ack 1390 win 7504 12:58:40.904061 IP 10.22.200.21.1228 > 10.20.10.141.ssh: . ack 2690 win 17072 12:58:40.942770 IP 10.20.10.141.ssh > 10.22.200.21.1228: P 2690:2738(48) ack 1390 win 7504 12:58:41.088509 IP 10.22.200.21.1228 > 10.20.10.141.ssh: . ack 2738 win 17024 Sorry in advance for not posting as a reply to the original message. I don't subscribe to the list. Just wanted to substantiate Volkers findings. Hope this helps, Matthew Grooms