From owner-freebsd-ipfw@freebsd.org Tue Oct 13 02:31:06 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 96C0D9D2F65 for ; Tue, 13 Oct 2015 02:31:06 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 40F7F1201 for ; Tue, 13 Oct 2015 02:31:05 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 2186A2C3230 for ; Tue, 13 Oct 2015 12:22:09 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id LM2MXX6ExgjG for ; Tue, 13 Oct 2015 12:22:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id E3A852C3232 for ; Tue, 13 Oct 2015 12:22:08 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 1cqVwm_-RVFD for ; Tue, 13 Oct 2015 12:22:08 +1000 (AEST) Received: from [192.168.156.153] (reddog2.lnk.telstra.net [110.142.196.96]) by mail.7sq.com.au (Postfix) with ESMTPSA id 9BCFB2C3230 for ; Tue, 13 Oct 2015 12:22:08 +1000 (AEST) From: Nathan Aherne Subject: Kernel NAT issues Message-Id: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> Date: Tue, 13 Oct 2015 12:23:52 +1000 To: freebsd-ipfw@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 02:31:06 -0000 I sent through a question to this list a little while ago and have been = trying to get IPFW NAT working since then. I have had some success but = not the success I need, everything is working correctly except NAT rules = for my particular use case.=20 I have read every Google result on the first 50 pages when searching for = =E2=80=9CIPFW NAT=E2=80=9D or =E2=80=9CIPFW kernel NAT=E2=80=9D. I would = really appreciate it if someone could help me out. My use case is as follows: 1. I need to use hairpin NAT - I am using Jails behind a http proxy and = some jails need to be able to communicate with each other but only over = the WAN IP. This is why I have not use PF. 2. Some jails need to be able to communicate with each other on the = private interface (lo1) 3. IPFW is configured as default deny 4. Each jail has a list of allowed ports for incoming and outgoing = connections, these are set on the jails private IP (10.0.0.0/16) 5. I am using a stateful firewall. At the moment I am testing my IPFW ruleset using =E2=80=9Chost = google.com =E2=80=9D I can see the traffic leave the = Jail, get natted, the response come back from 8.8.8.8 and the traffic is = then denied. It seems like the state is not being checked or my rules = are in the wrong place. I feel that I should be able to fix this but I = am obviously misunderstanding is how NAT works.=20 I was under the assumption that traffic flowed like this: 1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for = public IP, the traffic is natted, it goes out the WAN interface, comes = back, is natted and switched to lo1 interface, state is checked and it = passes as returning traffic. 2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for = private IP, the traffic is not natted, it stays on the lo1 interface and = goes directly to the 10.0.0.2 Jail. I know I could answer my last question if =E2=80=9CI read the code=E2=80=9D= and I have tried but am not getting it. Is my understanding of IPFW = kernel NAT correct? Regards, Nathan From owner-freebsd-ipfw@freebsd.org Tue Oct 13 03:37:50 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F516A115DD for ; Tue, 13 Oct 2015 03:37:50 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A7AC81CF7 for ; Tue, 13 Oct 2015 03:37:48 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id t9D3bfj7011057; Tue, 13 Oct 2015 14:37:41 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Tue, 13 Oct 2015 14:37:41 +1100 (EST) From: Ian Smith To: Nathan Aherne cc: freebsd-ipfw@freebsd.org Subject: Re: Kernel NAT issues In-Reply-To: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> Message-ID: <20151013142301.B67283@sola.nimnet.asn.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 03:37:50 -0000 On Tue, 13 Oct 2015 12:33:52 +1000, Nathan Aherne wrote: > I sent through a question to this list a little while ago and have > been trying to get IPFW NAT working since then. I have had some > success but not the success I need, everything is working correctly > except NAT rules for my particular use case. Unfortunately the rest of your message failed to quote properly here, i.e not quoted indented as above, so I'll leave it out for now; perhaps it's my old mailer (pine) at fault. Maybe plain ASCII text would help. That said, without sharing your actual ruleset with us, sanitised if need be, it seems unlikely that anyone will be able to work out what might be happening here solely from your textual description. cheers, Ian From owner-freebsd-ipfw@freebsd.org Tue Oct 13 03:50:09 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72BE5A11A52 for ; Tue, 13 Oct 2015 03:50:09 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BE5BC6D6 for ; Tue, 13 Oct 2015 03:50:08 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id BAE962C3232; Tue, 13 Oct 2015 13:48:20 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id QSlXnMkNLpGQ; Tue, 13 Oct 2015 13:48:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 884852C3233; Tue, 13 Oct 2015 13:48:20 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id QUDl8Y-oQR9t; Tue, 13 Oct 2015 13:48:20 +1000 (AEST) Received: from [192.168.156.153] (reddog2.lnk.telstra.net [110.142.196.96]) by mail.7sq.com.au (Postfix) with ESMTPSA id 204E42C3232; Tue, 13 Oct 2015 13:48:20 +1000 (AEST) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Kernel NAT issues From: Nathan Aherne In-Reply-To: <20151013142301.B67283@sola.nimnet.asn.au> Date: Tue, 13 Oct 2015 13:50:04 +1000 Cc: freebsd-ipfw@freebsd.org Message-Id: References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> To: Ian Smith X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 03:50:09 -0000 Hi Ian, Thank you for your response. I didn=E2=80=99t post my ruleset because I should be able to fix the = issue myself but I see now that my request to explain =E2=80=9Chow NAT = works=E2=80=9D was incorrect. I have now included my ruleset below (as well as my initial email). # Enable NAT ipfw nat 1 config ip $jip same_ports log 00005 allow ip from any to any via lo0 00006 deny ip from any to not me in via bce0 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 00101 check-state 00110 allow icmp from any to WWW.XXX.YYY .ZZZ recv = bce0 keep-state 00111 allow tcp from any to WWW.XXX.YYY .ZZZ = dst-port 65222 recv bce0 setup keep-state 00112 allow icmp from WWW.XXX.YYY .ZZZ to any xmit = bce0 keep-state 00113 allow tcp from WWW.XXX.YYY .ZZZ to any = dst-port 53,80,443,22,65222 xmit bce0 setup keep-state 00114 allow udp from WWW.XXX.YYY .ZZZ to any = dst-port 53,123 xmit bce0 keep-state 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup = keep-state 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = setup keep-state 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = keep-state 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup = keep-state 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup = keep-state 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup = keep-state 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup = keep-state 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state 65500 deny log ip from any to any 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = keep-state 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state 65534 deny log ip from any to any 65535 deny ip from any to any = **************************************************************************= ************ I sent through a question to this list a little while ago and have been = trying to get IPFW NAT working since then. I have had some success but = not the success I need, everything is working correctly except NAT rules = for my particular use case.=20 I have read every Google result on the first 50 pages when searching for = =E2=80=9CIPFW NAT=E2=80=9D or =E2=80=9CIPFW kernel NAT=E2=80=9D. I would = really appreciate it if someone could help me out. My use case is as follows: 1. I need to use hairpin NAT - I am using Jails behind a http proxy and = some jails need to be able to communicate with each other but only over = the WAN IP. This is why I have not use PF. 2. Some jails need to be able to communicate with each other on the = private interface (lo1) 3. IPFW is configured as default deny 4. Each jail has a list of allowed ports for incoming and outgoing = connections, these are set on the jails private IP (10.0.0.0/16) 5. I am using a stateful firewall. At the moment I am testing my IPFW ruleset using =E2=80=9Chost = google.com >=E2=80=9D I can see the traffic leave the Jail, get = natted, the response come back from 8.8.8.8 and the traffic is then = denied. It seems like the state is not being checked or my rules are in = the wrong place. I feel that I should be able to fix this but I am = obviously misunderstanding is how NAT works.=20 I was under the assumption that traffic flowed like this: 1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for = public IP, the traffic is natted, it goes out the WAN interface, comes = back, is natted and switched to lo1 interface, state is checked and it = passes as returning traffic. 2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for = private IP, the traffic is not natted, it stays on the lo1 interface and = goes directly to the 10.0.0.2 Jail. I know I could answer my last question if =E2=80=9CI read the code=E2=80=9D= and I have tried but am not getting it. Is my understanding of IPFW = kernel NAT correct? Regards, Nathan _______________________________________________ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw = To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org = =E2=80=9D = **************************************************************************= ************ Regards, Nathan > On 13 Oct 2015, at 1:37 pm, Ian Smith wrote: >=20 > On Tue, 13 Oct 2015 12:33:52 +1000, Nathan Aherne wrote: >=20 >> I sent through a question to this list a little while ago and have=20 >> been trying to get IPFW NAT working since then. I have had some=20 >> success but not the success I need, everything is working correctly=20= >> except NAT rules for my particular use case. >=20 > Unfortunately the rest of your message failed to quote properly here,=20= > i.e not quoted indented as above, so I'll leave it out for now; = perhaps=20 > it's my old mailer (pine) at fault. Maybe plain ASCII text would = help. >=20 > That said, without sharing your actual ruleset with us, sanitised if=20= > need be, it seems unlikely that anyone will be able to work out what=20= > might be happening here solely from your textual description. >=20 > cheers, Ian From owner-freebsd-ipfw@freebsd.org Tue Oct 13 05:57:59 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA3CFA1205D for ; Tue, 13 Oct 2015 05:57:58 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A52810AE for ; Tue, 13 Oct 2015 05:57:57 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 8B8C62C3230 for ; Tue, 13 Oct 2015 15:56:05 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id oSKpRIzroXBn for ; Tue, 13 Oct 2015 15:56:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 40F532C3231 for ; Tue, 13 Oct 2015 15:56:05 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id OZA_oavXAMFX for ; Tue, 13 Oct 2015 15:56:04 +1000 (AEST) Received: from [192.168.156.153] (reddog2.lnk.telstra.net [110.142.196.96]) by mail.7sq.com.au (Postfix) with ESMTPSA id CA4B92C3230 for ; Tue, 13 Oct 2015 15:56:04 +1000 (AEST) From: Nathan Aherne Message-Id: <5B1C303D-49F6-4EC2-B5B1-5F5D6BE8D4BE@reddog.com.au> Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Kernel NAT issues Date: Tue, 13 Oct 2015 15:57:49 +1000 In-Reply-To: Cc: freebsd-ipfw@freebsd.org References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 05:57:59 -0000 To further illustrate my issue, this is a small log output. I am running =E2=80=9Chost google.com =E2=80=9D in = the jail, which has the IP 10.0.0.1. The UNKNOWN line is logging on the = check-state rule. I would expect the first piece of traffic out would be = UNKNOWN (does not have an entry in the state table) but it seems the = returning traffic is also showing as UNKNOWN (the second 101). You can = see that the traffic is returning on the same port it went out on, so = its obviously the returning traffic. I am not sure why state is not = being kept? Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 Regards, Nathan > On 13 Oct 2015, at 1:50 pm, Nathan Aherne = wrote: >=20 > Hi Ian, >=20 > Thank you for your response. >=20 > I didn=E2=80=99t post my ruleset because I should be able to fix the = issue myself but I see now that my request to explain =E2=80=9Chow NAT = works=E2=80=9D was incorrect. >=20 > I have now included my ruleset below (as well as my initial email). >=20 > # Enable NAT > ipfw nat 1 config ip $jip same_ports log >=20 >=20 > 00005 allow ip from any to any via lo0 > 00006 deny ip from any to not me in via bce0 > 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 > 00101 check-state > 00110 allow icmp from any to WWW.XXX.YYY .ZZZ = recv bce0 keep-state > 00111 allow tcp from any to WWW.XXX.YYY .ZZZ = dst-port 65222 recv bce0 setup keep-state > 00112 allow icmp from WWW.XXX.YYY .ZZZ to any = xmit bce0 keep-state > 00113 allow tcp from WWW.XXX.YYY .ZZZ to any = dst-port 53,80,443,22,65222 xmit bce0 setup keep-state > 00114 allow udp from WWW.XXX.YYY .ZZZ to any = dst-port 53,123 xmit bce0 keep-state > 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup = keep-state > 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 = keep-state > 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 setup keep-state > 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 keep-state > 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup = keep-state > 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup = keep-state > 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state > 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup = keep-state > 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup = keep-state > 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state > 65500 deny log ip from any to any > 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = keep-state > 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state > 65534 deny log ip from any to any > 65535 deny ip from any to any >=20 > = **************************************************************************= ************ > I sent through a question to this list a little while ago and have = been trying to get IPFW NAT working since then. I have had some success = but not the success I need, everything is working correctly except NAT = rules for my particular use case.=20 >=20 > I have read every Google result on the first 50 pages when searching = for =E2=80=9CIPFW NAT=E2=80=9D or =E2=80=9CIPFW kernel NAT=E2=80=9D. I = would really appreciate it if someone could help me out. >=20 > My use case is as follows: >=20 > 1. I need to use hairpin NAT - I am using Jails behind a http proxy = and some jails need to be able to communicate with each other but only = over the WAN IP. This is why I have not use PF. > 2. Some jails need to be able to communicate with each other on the = private interface (lo1) > 3. IPFW is configured as default deny > 4. Each jail has a list of allowed ports for incoming and outgoing = connections, these are set on the jails private IP (10.0.0.0/16) > 5. I am using a stateful firewall. >=20 > At the moment I am testing my IPFW ruleset using =E2=80=9Chost = google.com >=E2=80=9D I can see the traffic leave the Jail, get = natted, the response come back from 8.8.8.8 and the traffic is then = denied. It seems like the state is not being checked or my rules are in = the wrong place. I feel that I should be able to fix this but I am = obviously misunderstanding is how NAT works.=20 >=20 > I was under the assumption that traffic flowed like this: >=20 > 1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is = for public IP, the traffic is natted, it goes out the WAN interface, = comes back, is natted and switched to lo1 interface, state is checked = and it passes as returning traffic. >=20 > 2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is = for private IP, the traffic is not natted, it stays on the lo1 interface = and goes directly to the 10.0.0.2 Jail. >=20 > I know I could answer my last question if =E2=80=9CI read the code=E2=80= =9D and I have tried but am not getting it. Is my understanding of IPFW = kernel NAT correct? >=20 > Regards, >=20 > Nathan >=20 > _______________________________________________ > freebsd-ipfw@freebsd.org mailing = list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw = > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org = =E2=80=9D >=20 > = **************************************************************************= ************ >=20 > Regards, >=20 > Nathan >=20 >> On 13 Oct 2015, at 1:37 pm, Ian Smith wrote: >>=20 >> On Tue, 13 Oct 2015 12:33:52 +1000, Nathan Aherne wrote: >>=20 >>> I sent through a question to this list a little while ago and have=20= >>> been trying to get IPFW NAT working since then. I have had some=20 >>> success but not the success I need, everything is working correctly=20= >>> except NAT rules for my particular use case. >>=20 >> Unfortunately the rest of your message failed to quote properly here,=20= >> i.e not quoted indented as above, so I'll leave it out for now; = perhaps=20 >> it's my old mailer (pine) at fault. Maybe plain ASCII text would = help. >>=20 >> That said, without sharing your actual ruleset with us, sanitised if=20= >> need be, it seems unlikely that anyone will be able to work out what=20= >> might be happening here solely from your textual description. >>=20 >> cheers, Ian >=20 > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to = "freebsd-ipfw-unsubscribe@freebsd.org" From owner-freebsd-ipfw@freebsd.org Wed Oct 14 09:06:34 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C3FDA0E589 for ; Wed, 14 Oct 2015 09:06:34 +0000 (UTC) (envelope-from anonymous@ip-72-167-142-123.ip.secureserver.net) Received: from p3plsmtps2ded01.prod.phx3.secureserver.net (p3plsmtps2ded01.prod.phx3.secureserver.net [208.109.80.58]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "Bizanga Labs SMTP Client Certificate", Issuer "Bizanga Labs CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D515215DE for ; Wed, 14 Oct 2015 09:06:33 +0000 (UTC) (envelope-from anonymous@ip-72-167-142-123.ip.secureserver.net) Received: from ip-72-167-142-123.ip.secureserver.net ([72.167.142.123]) by p3plsmtps2ded01.prod.phx3.secureserver.net with : DED : id Ux6W1r0532fwyUQ01x6W7r; Wed, 14 Oct 2015 02:06:32 -0700 x-originating-ip: 72.167.142.123 x_spam_cmae: v=2.1 cv=UdjfSciN c=1 sm=1 tr=0 p=S_VOcfIJLpEA:10 a=CUEiVRIz4OCDbXoRCWzYTw==:117 a=TZb1taSUAAAA:8 a=hGqpr2WsAAAA:8 a=5lJygRwiOn0A:10 a=twKErMXSE2DJwNbQDlgA:9 a=Xp4i6JP7Cu6tkGgS:21 a=v9oboHvABcBdHrh4:21 a=CjuIK1q_8ugA:10 a=rHjJoKSNxMMe0KSbmIgA:9 a=IKIoO-ieCDEA:10 Received: (qmail 28782 invoked by uid 10001); 11 Oct 2015 17:57:37 -0700 To: freebsd-ipfw@freebsd.org Subject: Problems with item delivery, n.0000749477 X-PHP-Originating-Script: 10001:post.php(3) : regexp code(1) : eval()'d code(11) : eval()'d code Date: Sun, 11 Oct 2015 18:57:37 -0600 From: "FedEx 2Day A.M." Reply-To: "FedEx 2Day A.M." Message-ID: X-Priority: 3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2015 09:06:34 -0000 Dear Customer, Courier was unable to deliver the parcel to you. Shipment Label is attached to email. Yours trully, Rene Little, FedEx Operation Agent. From owner-freebsd-ipfw@freebsd.org Wed Oct 14 14:51:40 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9BC57A13B47 for ; Wed, 14 Oct 2015 14:51:40 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9E7ACC75 for ; Wed, 14 Oct 2015 14:51:38 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id t9EEpUDh084592; Thu, 15 Oct 2015 01:51:30 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Thu, 15 Oct 2015 01:51:30 +1100 (EST) From: Ian Smith To: Nathan Aherne cc: freebsd-ipfw@freebsd.org Subject: Re: Kernel NAT issues In-Reply-To: Message-ID: <20151014232026.S15983@sola.nimnet.asn.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2015 14:51:40 -0000 On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote: > Hi Ian, > > Thank you for your response. > > I didn˙˙t post my ruleset because I should be able to fix the issue > myself but I see now that my request to explain ˙˙how NAT works˙˙ was > incorrect. > > I have now included my ruleset below (as well as my initial email). Hi Nathan, I was really hoping someone who knows more about stateful rule handling (and jail networking) might have a go at this. Oh well I'll try, but I'm a lousy mindreader, and really don't know which of the below constitutes 'hairpin NAT'. Perhaps showing your 'netstat -finet -an' and 'netstat -finet -rn' may shed light on routing? And 'ifconfig'? > # Enable NAT > ipfw nat 1 config ip $jip same_ports log I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed to you, and so traffic to it won't be subject to NAT .. correct? But the WWW... address and all 10.0/16 addresses are jails, not any separate boxes you gateway for, right? Just the one external interface, right? > 00005 allow ip from any to any via lo0 > 00006 deny ip from any to not me in via bce0 > 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 > 00101 check-state Ok, inbound from WAN is nat'd and existing stateful flows followed by executing the rule that originally kept state. Where this is a skipto, skipto will be performed. But where it's a nat rule, I've no idea .. see below, but you really don't want to add keep-state (again) there. > 00110 allow icmp from any to WWW.XXX.YYY .ZZZ recv bce0 keep-state Hmm. I'd limit this to perhaps icmptypes 0,3,8,11 - though a stateless rule would make more sense especially for inbound ICMP. But moving on .. > 00111 allow tcp from any to WWW.XXX.YYY .ZZZ dst-port 65222 recv bce0 setup keep-state Ok, but showting why plain text works better than HTML on lists :) > 00112 allow icmp from WWW.XXX.YYY .ZZZ to any xmit bce0 keep-state > 00113 allow tcp from WWW.XXX.YYY .ZZZ to any dst-port 53,80,443,22,65222 xmit bce0 setup keep-state > 00114 allow udp from WWW.XXX.YYY .ZZZ to any dst-port 53,123 xmit bce0 keep-state Smells ok. > 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state > 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's got a problem .. see below. These two are inbound traffic (recv) and as is, skipping to 65501 will fall through two outbound rules to be denied. Either allow them here directly, or likely better, skipto a separate target that then allows (or denies) them, if that's what you intended? > 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 setup keep-state > 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state Ok, this traffic does needs to be NAT'd on the way out. > 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup keep-state > 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup keep-state > 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state Not clear why these tcp ports are open inbound and outbound? Presumably this is jail-to-jail traffic? Perhaps not relevant to your problem. > 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup keep-state > 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup keep-state > 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state > 65500 deny log ip from any to any Ok. > 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state This the target for outbound traffix, xmit bce0, so nat is appropriate. Does jail-to-jail traffic travels via lo1? Or what? This won't do anything to inbound traffic, but that really shouldn't get here except returns as the result of check-state - not from 120 & 121. But keep-state is not ok, state was already set on the skipto. I don't know how this extra keep-state might behave - does anyone have an idea? Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic. > 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state So, only remaining traffic is outbound from the host itself, and traffic that is to 10.0/16, but not from AAA... is to be dropped, correct? I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp states .. myself, I'd go for separate rules for each eg tcp, udp, .. and I'd do it somewhere else than as a fall through from outbound nat rule, it's confusing here, to me anyway .. unless I've missed the reason? > 65534 deny log ip from any to any > 65535 deny ip from any to any Ok, now for your demo of the problem from the later mail, which I've reformated to quote properly, so: > To further illustrate my issue, this is a small log output. > > I am running host google.com in the jail, which > has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state > rule. I see you don't have logging on 101 above now. Probably best. > I would expect the first piece of traffic out would be UNKNOWN > (does not have an entry in the state table) but it seems the > returning traffic is also showing as UNKNOWN (the second 101). I've never logged a check-state, but UNKNOWN may not mean that .. > You can see that the traffic is returning on the same port it went > out on, so its obviously the returning traffic. I am not sure why > state is not being kept? Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, so it's been correctly NAT'd on the way in. Get rid of that keep-state on the nat rule at 65501 and see if not creating double entries in the state table helps. And change the skipto target on 120 & 121 to only pass outbound traffic to outbound NAT rule/s. Once you've done outbound NAT, probably best just to 'allow [log] all'? > Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0 > Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0 > Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0 > Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0 > Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0 > Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0 That said, I can see why this return packet would be denied even if it were in the nat table: it would execute 'skipto 65501', which nat rule does not apply, as it's not outbound, and rule 65502 does not apply, as it's neither from AAA... nor outbound, so it's then denied by 65534. Hope this helps. Please cc me on any response to the list. It would be great if someone else might care to lend an oar here; I'm paddling out of my depth. cheers, Ian [..]