From owner-freebsd-ipfw@freebsd.org Tue Oct 13 05:57:59 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA3CFA1205D for ; Tue, 13 Oct 2015 05:57:58 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A52810AE for ; Tue, 13 Oct 2015 05:57:57 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 8B8C62C3230 for ; Tue, 13 Oct 2015 15:56:05 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id oSKpRIzroXBn for ; Tue, 13 Oct 2015 15:56:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 40F532C3231 for ; Tue, 13 Oct 2015 15:56:05 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id OZA_oavXAMFX for ; Tue, 13 Oct 2015 15:56:04 +1000 (AEST) Received: from [192.168.156.153] (reddog2.lnk.telstra.net [110.142.196.96]) by mail.7sq.com.au (Postfix) with ESMTPSA id CA4B92C3230 for ; Tue, 13 Oct 2015 15:56:04 +1000 (AEST) From: Nathan Aherne Message-Id: <5B1C303D-49F6-4EC2-B5B1-5F5D6BE8D4BE@reddog.com.au> Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Kernel NAT issues Date: Tue, 13 Oct 2015 15:57:49 +1000 In-Reply-To: Cc: freebsd-ipfw@freebsd.org References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 05:57:59 -0000 To further illustrate my issue, this is a small log output. I am running =E2=80=9Chost google.com =E2=80=9D in = the jail, which has the IP 10.0.0.1. The UNKNOWN line is logging on the = check-state rule. I would expect the first piece of traffic out would be = UNKNOWN (does not have an entry in the state table) but it seems the = returning traffic is also showing as UNKNOWN (the second 101). You can = see that the traffic is returning on the same port it went out on, so = its obviously the returning traffic. I am not sure why state is not = being kept? Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 Regards, Nathan > On 13 Oct 2015, at 1:50 pm, Nathan Aherne = wrote: >=20 > Hi Ian, >=20 > Thank you for your response. >=20 > I didn=E2=80=99t post my ruleset because I should be able to fix the = issue myself but I see now that my request to explain =E2=80=9Chow NAT = works=E2=80=9D was incorrect. >=20 > I have now included my ruleset below (as well as my initial email). >=20 > # Enable NAT > ipfw nat 1 config ip $jip same_ports log >=20 >=20 > 00005 allow ip from any to any via lo0 > 00006 deny ip from any to not me in via bce0 > 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 > 00101 check-state > 00110 allow icmp from any to WWW.XXX.YYY .ZZZ = recv bce0 keep-state > 00111 allow tcp from any to WWW.XXX.YYY .ZZZ = dst-port 65222 recv bce0 setup keep-state > 00112 allow icmp from WWW.XXX.YYY .ZZZ to any = xmit bce0 keep-state > 00113 allow tcp from WWW.XXX.YYY .ZZZ to any = dst-port 53,80,443,22,65222 xmit bce0 setup keep-state > 00114 allow udp from WWW.XXX.YYY .ZZZ to any = dst-port 53,123 xmit bce0 keep-state > 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup = keep-state > 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 = keep-state > 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 setup keep-state > 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 keep-state > 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup = keep-state > 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup = keep-state > 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state > 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup = keep-state > 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup = keep-state > 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state > 65500 deny log ip from any to any > 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = keep-state > 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state > 65534 deny log ip from any to any > 65535 deny ip from any to any >=20 > = **************************************************************************= ************ > I sent through a question to this list a little while ago and have = been trying to get IPFW NAT working since then. I have had some success = but not the success I need, everything is working correctly except NAT = rules for my particular use case.=20 >=20 > I have read every Google result on the first 50 pages when searching = for =E2=80=9CIPFW NAT=E2=80=9D or =E2=80=9CIPFW kernel NAT=E2=80=9D. I = would really appreciate it if someone could help me out. >=20 > My use case is as follows: >=20 > 1. I need to use hairpin NAT - I am using Jails behind a http proxy = and some jails need to be able to communicate with each other but only = over the WAN IP. This is why I have not use PF. > 2. Some jails need to be able to communicate with each other on the = private interface (lo1) > 3. IPFW is configured as default deny > 4. Each jail has a list of allowed ports for incoming and outgoing = connections, these are set on the jails private IP (10.0.0.0/16) > 5. I am using a stateful firewall. >=20 > At the moment I am testing my IPFW ruleset using =E2=80=9Chost = google.com >=E2=80=9D I can see the traffic leave the Jail, get = natted, the response come back from 8.8.8.8 and the traffic is then = denied. It seems like the state is not being checked or my rules are in = the wrong place. I feel that I should be able to fix this but I am = obviously misunderstanding is how NAT works.=20 >=20 > I was under the assumption that traffic flowed like this: >=20 > 1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is = for public IP, the traffic is natted, it goes out the WAN interface, = comes back, is natted and switched to lo1 interface, state is checked = and it passes as returning traffic. >=20 > 2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is = for private IP, the traffic is not natted, it stays on the lo1 interface = and goes directly to the 10.0.0.2 Jail. >=20 > I know I could answer my last question if =E2=80=9CI read the code=E2=80= =9D and I have tried but am not getting it. Is my understanding of IPFW = kernel NAT correct? >=20 > Regards, >=20 > Nathan >=20 > _______________________________________________ > freebsd-ipfw@freebsd.org mailing = list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw = > To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe@freebsd.org = =E2=80=9D >=20 > = **************************************************************************= ************ >=20 > Regards, >=20 > Nathan >=20 >> On 13 Oct 2015, at 1:37 pm, Ian Smith wrote: >>=20 >> On Tue, 13 Oct 2015 12:33:52 +1000, Nathan Aherne wrote: >>=20 >>> I sent through a question to this list a little while ago and have=20= >>> been trying to get IPFW NAT working since then. I have had some=20 >>> success but not the success I need, everything is working correctly=20= >>> except NAT rules for my particular use case. >>=20 >> Unfortunately the rest of your message failed to quote properly here,=20= >> i.e not quoted indented as above, so I'll leave it out for now; = perhaps=20 >> it's my old mailer (pine) at fault. Maybe plain ASCII text would = help. >>=20 >> That said, without sharing your actual ruleset with us, sanitised if=20= >> need be, it seems unlikely that anyone will be able to work out what=20= >> might be happening here solely from your textual description. >>=20 >> cheers, Ian >=20 > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to = "freebsd-ipfw-unsubscribe@freebsd.org"