From owner-freebsd-net@FreeBSD.ORG Sat May 6 21:15:46 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C82016A406 for ; Sat, 6 May 2006 21:15:46 +0000 (UTC) (envelope-from dimas@dataart.com) Received: from relay1.dataart.com (fobos.marketsite.ru [62.152.84.30]) by mx1.FreeBSD.org (Postfix) with ESMTP id 950F743D45 for ; Sat, 6 May 2006 21:15:44 +0000 (GMT) (envelope-from dimas@dataart.com) Received: from e1.universe.dart.spb ([192.168.10.44]) by relay1.dataart.com with esmtp (Exim 4.22) id 1FcU7q-0003au-Rf for freebsd-net@freebsd.org; Sun, 07 May 2006 01:15:42 +0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Sun, 7 May 2006 01:14:13 +0400 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: IPSEC tunnel problem thread-index: AcZvnMHHnY+yeEvlQp681OnHEpDCeQAJV1LgABvjK2AASAjcUA== From: "Dmitry Andrianov" To: Subject: FW: IPSEC tunnel problem X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 May 2006 21:15:46 -0000 Hello freebsd-net, I had a problem with IPSEC which is actually already solved on freebsd-pf thanks to list members. I'm forwarding "a solution" here because when I was google-ing for a solution I found similar problems on this list as well. And it looks like these are still unanswered. Examples are: IPSec session stalls ( http://lists.freebsd.org/pipermail/freebsd-net/2005-October/008745.html ) more on IPSec + gif stalling ( http://lists.freebsd.org/pipermail/freebsd-net/2005-October/008812.html ) Also there are still three issues I would like someone experienced take a look at before I start filling PRs :=3D) Regards, Dmitry Andrianov=20 -----Original Message----- From: Dmitry Andrianov=20 Sent: Friday, May 05, 2006 4:06 PM To: 'Daniel Hartmeier' Cc: 'freebsd-pf@freebsd.org' Subject: RE: IPSEC tunnel problem Ok, finally, IPSEC_FILTERGIF fixes the problem with stalling connections. Daniel, thanks a lot for your help. However I see at least three issues which needs to be handled somehow: 1. I believe that these days FreeBSD+pf becomes very popular configuration and the fact that system does not work out of the box in this configuration and requires some tricks is not a good sign. My personal opinion is that IPSEC_FILTERGIF just should be on by default. Or at least IPSEC section of FreeBSD handbook should say something about it. Making kernel option on by default has advantage because there are many places with IPSEC how-to on Internet and it is not possible to fix all of them. So the question is should I fill doc/* PR or kernel/* PR for that? 2. Why the router replies with ICMP host-unreachable to the TCP packets with wrong state if block policy is set to "drop" and not "return"? 3. Below is the dump of packets how pf sees 3way handshake with IPSEC_FILTERGIF option 15:22:53.192253 rule 0/0(match): pass in on fxp0: 10.1.0.1 > 10.1.0.2: ESP(spi=3D0x00002708,seq=3D0x9b05), length 104 15:22:53.192267 rule 0/0(match): pass in on fxp0: 10.1.0.1 > 10.1.0.2: 192.168.10.176.2169 > 10.2.0.2.3389: S 327999972:327999972(0) win 32768 (ipip-proto-4) 15:22:53.192276 rule 0/0(match): pass in on gif0: 192.168.10.176.2169 > 10.2.0.2.3389: S 327999972:327999972(0) win 32768 15:22:53.192286 rule 0/0(match): pass out on fxp1: 192.168.10.176.2169 > 10.2.0.2.3389: S 327999972:327999972(0) win 32768 15:22:53.192436 rule 0/0(match): pass in on fxp1: 10.2.0.2.3389 > 192.168.10.176.2169: S 1559792967:1559792967(0) ack 327999973 win 65535 15:22:53.192445 rule 0/0(match): pass out on gif0: 10.2.0.2.3389 > 192.168.10.176.2169: S 1559792967:1559792967(0) ack 327999973 win 65535 15:22:53.192460 rule 0/0(match): pass out on fxp0: 10.1.0.2 > 10.1.0.1: ESP(spi=3D0x00002707,seq=3D0x18a4), length 104 15:22:53.193098 rule 0/0(match): pass in on fxp0: 10.1.0.1 > 10.1.0.2: ESP(spi=3D0x00002708,seq=3D0x9b06), length 96 15:22:53.193108 rule 0/0(match): pass in on fxp0: 10.1.0.1 > 10.1.0.2: 192.168.10.176.2169 > 10.2.0.2.3389: . ack 1 win 33580 (ipip-proto-4) 15:22:53.193116 rule 0/0(match): pass in on gif0: 192.168.10.176.2169 > 10.2.0.2.3389: . ack 1 win 33580 15:22:53.193123 rule 0/0(match): pass out on fxp1: 192.168.10.176.2169 > 10.2.0.2.3389: . ack 1 win 33580 Clearly, pf sees each packet coming from the tunnel as four (ESP in on WAN interface, ipencap in on WAN interface, in on gif and out packet on LAN interface) while only three packets for data coming from LAN to tunnel. For me this has two disadvantages: * I have to allow ipencap between endpoint in firewall, otherwise decapulated IPSEC packet gets blocked. I do not like this thing because it will also allow someone sending ipencap packets with spoofed source easily. * It is just a bit inconsistent - I would prefer either seeing none of ipencaps (in and out) or seeing them both. I could fill PR about that one too, but I'm not sure what is the best way to propose. Regards, Dmitry Andrianov -----Original Message----- From: Dmitry Andrianov Sent: Friday, May 05, 2006 1:42 AM To: 'Daniel Hartmeier' Cc: freebsd-pf@freebsd.org Subject: RE: IPSEC tunnel problem Daniel, Looks like you are right. Below is dump from pflog0 interface of 3way handshake to 3389: 01:27:55.888031 rule 0/0(match): pass out on fxp1: 192.168.10.176.3809 > 10.2.0.2.3389: S 220200183:220200183(0) win 32768 01:27:55.888177 rule 0/0(match): pass in on fxp1: 10.2.0.2.3389 > 192.168.10.176.3809: S 1690303870:1690303870(0) ack 220200184 win 65535 01:27:55.888186 rule 0/0(match): pass out on gif0: 10.2.0.2.3389 > 192.168.10.176.3809: S 1690303870:1690303870(0) ack 220200184 win 65535 01:27:55.888884 rule 0/0(match): pass out on fxp1: 192.168.10.176.3809 > 10.2.0.2.3389: . ack 1 win 33580 Packets coming from the VPN only appear as "out" packets on LAN ethernet. While packets coming from LAN appear as "in" packets on ethernet and "out" packets on gif. I assume this is because my kernel was not build with=20 # Set IPSEC_FILTERGIF to force packets coming through a gif tunnel # to be processed by any configured packet filtering (ipfw, ipf). # The default is that packets coming from a tunnel are _not_ processed; # they are assumed trusted. # # IPSEC history is preserved for such packets, and can be filtered # using ipfw(8)'s 'ipsec' keyword, when this option is enabled. # #options IPSEC_FILTERGIF #filter ipsec packets from a tunnel Well. I knew about this option before but I thought it makes sense only when I want filtering packets on gif0 which I currently do not. I never realized absence of that option may break something else as happened in my case. Tomorrow will rebuild kernel and re-run my tests. To me things like that are subject for FAQ or even better - should be on by default. IMHO. Regards, Dmitry Andrianov =20 -----Original Message----- From: Daniel Hartmeier [mailto:daniel@benzedrine.cx] Sent: Thursday, May 04, 2006 8:40 PM To: Dmitry Andrianov Cc: freebsd-pf@freebsd.org Subject: Re: IPSEC tunnel problem On Thu, May 04, 2006 at 08:03:55PM +0400, Dmitry Andrianov wrote: > May 4 19:52:53 vrn1 kernel: pf: BAD state: TCP 10.2.0.2:3389 > 10.2.0.2:3389 192.168.10.100:1919 [lo=3D4162748520 high=3D4162681620 > win=3D65535 modulator=3D0] [lo=3D0 high=3D65535 win=3D1 modulator=3D0] = 2:0 PA=20 > seq=3D4162748520 ack=3D0 len=3D632 ackskew=3D0 pkts=3D245:0 = dir=3Dout,fwd The 'dir=3Dout,fwd' part means that the state was created from a packet going out on the interface (gif0, I assume), and that the packet being blocked here was in the same direction. The 'pkts=3D245:0' part means that the state entry has so far matched = 245 packets flowing in the same direction (out), but 0 in the reverse direction (in). And that's the problem, pf is not associating replies with the state entry. Because of that, the state entry does not advance its sequence number window (advertised in the replies), and eventually stalls the connection. This is probably related to the gif interface. I haven't tried it on FreeBSD, but for stateful filtering, it would be important that pf sees packets in both directions on that interface (i.e. SYN outgoing, SYN+ACK incoming, etc.) You can test what packets pf sees in what direction on an interface by replacing the ruleset with a single rule like pass log all and observing pflog0 (with tcpdump, for instance) while establishing a connection. If only packets of one direction are seen (or both outgoing and incoming packets are seen as having the same direction), there might be a problem with pfil hooks in gif. Daniel