From owner-freebsd-net@FreeBSD.ORG Mon Sep 11 21:22:47 2006 Return-Path: X-Original-To: net@freebsd.org Delivered-To: freebsd-net@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3FC1516A40F for ; Mon, 11 Sep 2006 21:22:47 +0000 (UTC) (envelope-from kbyanc@posi.net) Received: from ylpvm12.prodigy.net (ylpvm12-ext.prodigy.net [207.115.57.43]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0C1A743D4C for ; Mon, 11 Sep 2006 21:22:04 +0000 (GMT) (envelope-from kbyanc@posi.net) X-ORBL: [70.231.128.103] Received: from gateway.posi.net (adsl-70-231-128-103.dsl.snfc21.sbcglobal.net [70.231.128.103]) by ylpvm12.prodigy.net (8.13.7 out spool5000 dk/8.13.7) with ESMTP id k8BLLRSc029021; Mon, 11 Sep 2006 17:21:28 -0400 Received: from localhost (localhost [127.0.0.1]) by gateway.posi.net (Postfix) with ESMTP id 3660275E05F; Mon, 11 Sep 2006 14:22:00 -0700 (PDT) Date: Mon, 11 Sep 2006 14:22:00 -0700 (PDT) From: Kelly Yancey To: Eugene Grosbein In-Reply-To: <200609111341.k8BDfneZ020221@nkz.delikates-nk.ru> Message-ID: <20060911131513.S27693@gateway.posi.net> References: <200609111341.k8BDfneZ020221@nkz.delikates-nk.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: VANHULLEBUS Yvan , net@freebsd.org Subject: Re: ipsec with ipfw divert (not NAT) encodes a packet twice breaking PMTUD X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2006 21:22:47 -0000 On Mon, 11 Sep 2006, Eugene Grosbein wrote: > > >Submitter-Id: current-users > >Originator: Eugene Grosbein > >Organization: Svyaz Service JSC > >Confidential: no > >Synopsis: ipsec with ipfw divert (not NAT) encodes a packet twice breaking PMTUD > >Severity: serious > >Priority: high > >Category: kern > >Class: sw-bug > >Release: FreeBSD 6.1-STABLE i386 > >Environment: > System: FreeBSD nkz.delikates-nk.ru 6.1-STABLE FreeBSD 6.1-STABLE #1: Thu Sep 7 13:31:53 KRAST 2006 root@nkz.delikates-nk.ru:/home/obj/home/src/sys/NKZ i386 > options IPDIVERT > options IPSEC > options IPSEC_ESP > > >Description: > When outgoing packet encoded due to corresponding IPSEC policy > is passed to divert socket (f.e. to ipacctd for accounting), > it is encoded second time with IPSEC then. Besides obvious > logic error, this also results in broken Path MTU Discovery. > > >How-To-Repeat: > > Use a kernel with options IPDIVERT, IPSEC, IPSEC_ESP > (my kernel also contains IPSEC_FILTERGIF, but this should not matter). > > Suppose there are two local nets numbered 192.168.1.0/24 > and 192.168.2.0/24, each has a FreeBSD router > (192.168.1.1 and 192.168.2.1). Routers make gif(4) tunnel between > and use IPSEC transport mode to encrypt its contents. > Their external IP addresses are 1.1.1.1 and 2.2.2.2 > Just FYI, when we implemented the enc interface for FreeBSD 4.10 for one of our products at work, we encountered a similar issue. The problem is that you need to add a flag to the sockaddr_in passed to the divert(4) consumer; when that consumer re-injects the packets into the network stack, ip_output() needs to check for the flag and goto skip_ipsec to avoid re-encapsulation. The next issue is that there is no room in the sockaddr_in structure for such a flag. We resorted to a hack (eventually, we re-implemented the divert interface to have its own sockaddr_div rather than overloading sockaddr_in, but that is another story). We stuck the flag indicating whether to skip IPsec encapsulation on input in the high bit of the first byte in the sin_zero array. This only works because natd(8) doesn't inspect the (partial) interface name stored in the sockaddr_in's sin_zero array. "Hack" doesn't really being to describe the hideousness of this workaround, but it will get the job done. It looks like the same effect might be able to be achieved by modifying natd to be able to set the policy on the divert socket to IPSEC_POLICY_NONE via the IP_IPSEC_POLICY socket option, but I don't know enough about that code path to say for sure. Good luck, Kelly -- Kelly Yancey - kbyanc@posi.net - kelly@nttmcl.com