From owner-svn-src-stable-11@freebsd.org  Sun Apr 14 12:28:44 2019
Return-Path: <owner-svn-src-stable-11@freebsd.org>
Delivered-To: svn-src-stable-11@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C1021574741;
 Sun, 14 Apr 2019 12:28:44 +0000 (UTC) (envelope-from ae@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
 [IPv6:2610:1c1:1:606c::19:3])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 server-signature RSA-PSS (4096 bits)
 client-signature RSA-PSS (4096 bits) client-digest SHA256)
 (Client CN "mxrelay.nyi.freebsd.org",
 Issuer "Let's Encrypt Authority X3" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id B32088F57D;
 Sun, 14 Apr 2019 12:28:43 +0000 (UTC) (envelope-from ae@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
 [IPv6:2610:1c1:1:6068::e6a:0])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 88ADA206D1;
 Sun, 14 Apr 2019 12:28:43 +0000 (UTC) (envelope-from ae@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
 by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x3ECShev048117;
 Sun, 14 Apr 2019 12:28:43 GMT (envelope-from ae@FreeBSD.org)
Received: (from ae@localhost)
 by repo.freebsd.org (8.15.2/8.15.2/Submit) id x3ECSg66048109;
 Sun, 14 Apr 2019 12:28:42 GMT (envelope-from ae@FreeBSD.org)
Message-Id: <201904141228.x3ECSg66048109@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: ae set sender to ae@FreeBSD.org
 using -f
From: "Andrey V. Elsukov" <ae@FreeBSD.org>
Date: Sun, 14 Apr 2019 12:28:42 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
 svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org
Subject: svn commit: r346209 - in stable/11: sbin/ipfw sys/conf
 sys/modules/ipfw_nat64 sys/netpfil/ipfw/nat64
X-SVN-Group: stable-11
X-SVN-Commit-Author: ae
X-SVN-Commit-Paths: in stable/11: sbin/ipfw sys/conf sys/modules/ipfw_nat64
 sys/netpfil/ipfw/nat64
X-SVN-Commit-Revision: 346209
X-SVN-Commit-Repository: base
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Rspamd-Queue-Id: B32088F57D
X-Spamd-Bar: --
Authentication-Results: mx1.freebsd.org
X-Spamd-Result: default: False [-2.96 / 15.00];
 local_wl_from(0.00)[FreeBSD.org];
 NEURAL_HAM_MEDIUM(-1.00)[-0.999,0];
 NEURAL_HAM_LONG(-1.00)[-1.000,0];
 NEURAL_HAM_SHORT(-0.97)[-0.966,0];
 ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]
X-BeenThere: svn-src-stable-11@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: SVN commit messages for only the 11-stable src tree
 <svn-src-stable-11.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/svn-src-stable-11>, 
 <mailto:svn-src-stable-11-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-stable-11/>
List-Post: <mailto:svn-src-stable-11@freebsd.org>
List-Help: <mailto:svn-src-stable-11-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/svn-src-stable-11>, 
 <mailto:svn-src-stable-11-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 14 Apr 2019 12:28:44 -0000

Author: ae
Date: Sun Apr 14 12:28:41 2019
New Revision: 346209
URL: https://svnweb.freebsd.org/changeset/base/346209

Log:
  MFC r339542:
    Retire IPFIREWALL_NAT64_DIRECT_OUTPUT kernel option. And add ability
    to switch the output method in run-time. Also document some sysctl
    variables that can by changed for NAT64 module.
  
    NAT64 had compile time option IPFIREWALL_NAT64_DIRECT_OUTPUT to use
    if_output directly from nat64 module. By default is used netisr based
    output method. Now both methods can be used, but they require different
    handling by rules.
  
    Obtained from:	Yandex LLC
    Sponsored by:	Yandex LLC
    Differential Revision:	https://reviews.freebsd.org/D16647

Modified:
  stable/11/sbin/ipfw/ipfw.8
  stable/11/sys/conf/options
  stable/11/sys/modules/ipfw_nat64/Makefile
  stable/11/sys/netpfil/ipfw/nat64/ip_fw_nat64.c
  stable/11/sys/netpfil/ipfw/nat64/nat64_translate.c
  stable/11/sys/netpfil/ipfw/nat64/nat64_translate.h
Directory Properties:
  stable/11/   (props changed)

Modified: stable/11/sbin/ipfw/ipfw.8
==============================================================================
--- stable/11/sbin/ipfw/ipfw.8	Sun Apr 14 12:14:25 2019	(r346208)
+++ stable/11/sbin/ipfw/ipfw.8	Sun Apr 14 12:28:41 2019	(r346209)
@@ -3304,9 +3304,14 @@ Make sure that ND6 neighbor solicitation (ICMPv6 type 
 advertisement (ICMPv6 type 136) messages will not be handled by translation
 rules.
 .Pp
-After translation NAT64 translator sends packets through corresponding netisr
-queue.
+After translation NAT64 translator by default sends packets through
+corresponding netisr queue.
 Thus translator host should be configured as IPv4 and IPv6 router.
+Also this means, that a packet is handled by firewall twice.
+First time an original packet is handled and consumed by translator,
+and then it is handled again as translated packet.
+This behavior can be changed by sysctl variable 
+.Va net.inet.ip.fw.nat64_direct_output .
 .Pp
 The stateful NAT64 configuration command is the following:
 .Bd -ragged -offset indent
@@ -3929,6 +3934,41 @@ Default is no.
 Controls whether bridged packets are passed to
 .Nm .
 Default is no.
+.It Va net.inet.ip.fw.nat64_allow_private : No 0
+Defines how
+.Nm nat64
+handles private IPv4 addresses:
+.Bl -tag -width indent
+.It Cm 0
+Packets with private IPv4 will not be handled by translator
+.It Cm 1
+Translator will accept and process packets with private IPv4 addresses.
+.El
+.It Va net.inet.ip.fw.nat64_debug : No 0
+Controls debugging messages produced by
+.Nm ipfw_nat64
+module.
+.It Va net.inet.ip.fw.nat64_direct_output : No 0
+Controls the output method used by
+.Nm ipfw_nat64
+module:
+.Bl -tag -width indent
+.It Cm 0
+A packet is handled by 
+.Nm ipfw
+twice.
+First time an original packet is handled by
+.Nm ipfw
+and consumed by
+.Nm ipfw_nat64
+translator.
+Then translated packet is queued via netisr to input processing again.
+.It Cm 1
+A packet is handled by
+.Nm ipfw
+only once, and after translation it will be pushed directly to outgoing
+interface.
+.El
 .El
 .Sh INTERNAL DIAGNOSTICS
 There are some commands that may be useful to understand current state

Modified: stable/11/sys/conf/options
==============================================================================
--- stable/11/sys/conf/options	Sun Apr 14 12:14:25 2019	(r346208)
+++ stable/11/sys/conf/options	Sun Apr 14 12:28:41 2019	(r346209)
@@ -422,7 +422,6 @@ IPFIREWALL		opt_ipfw.h
 IPFIREWALL_DEFAULT_TO_ACCEPT	opt_ipfw.h
 IPFIREWALL_NAT		opt_ipfw.h
 IPFIREWALL_NAT64	opt_ipfw.h
-IPFIREWALL_NAT64_DIRECT_OUTPUT	opt_ipfw.h
 IPFIREWALL_NPTV6	opt_ipfw.h
 IPFIREWALL_VERBOSE	opt_ipfw.h
 IPFIREWALL_VERBOSE_LIMIT	opt_ipfw.h

Modified: stable/11/sys/modules/ipfw_nat64/Makefile
==============================================================================
--- stable/11/sys/modules/ipfw_nat64/Makefile	Sun Apr 14 12:14:25 2019	(r346208)
+++ stable/11/sys/modules/ipfw_nat64/Makefile	Sun Apr 14 12:28:41 2019	(r346209)
@@ -6,8 +6,5 @@ KMOD=	ipfw_nat64
 SRCS=	ip_fw_nat64.c nat64_translate.c
 SRCS+=	nat64lsn.c nat64lsn_control.c
 SRCS+=	nat64stl.c nat64stl_control.c
-SRCS+=	opt_ipfw.h
-
-#CFLAGS+=	-DIPFIREWALL_NAT64_DIRECT_OUTPUT
 
 .include <bsd.kmod.mk>

Modified: stable/11/sys/netpfil/ipfw/nat64/ip_fw_nat64.c
==============================================================================
--- stable/11/sys/netpfil/ipfw/nat64/ip_fw_nat64.c	Sun Apr 14 12:14:25 2019	(r346208)
+++ stable/11/sys/netpfil/ipfw/nat64/ip_fw_nat64.c	Sun Apr 14 12:28:41 2019	(r346209)
@@ -48,6 +48,7 @@ __FBSDID("$FreeBSD$");
 #include <netpfil/ipfw/ip_fw_private.h>
 
 #include "ip_fw_nat64.h"
+#include "nat64_translate.h"
 
 VNET_DEFINE(int, nat64_debug) = 0;
 VNET_DEFINE(int, nat64_allow_private) = 0;
@@ -56,8 +57,26 @@ SYSCTL_DECL(_net_inet_ip_fw);
 SYSCTL_INT(_net_inet_ip_fw, OID_AUTO, nat64_debug, CTLFLAG_VNET | CTLFLAG_RW,
     &VNET_NAME(nat64_debug), 0, "Debug level for NAT64 module");
 SYSCTL_INT(_net_inet_ip_fw, OID_AUTO, nat64_allow_private,
-    CTLFLAG_VNET |CTLFLAG_RW, &VNET_NAME(nat64_allow_private), 0,
+    CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(nat64_allow_private), 0,
     "Allow use of non-global IPv4 addresses with NAT64");
+
+static int
+sysctl_direct_output(SYSCTL_HANDLER_ARGS)
+{
+	uint32_t value;
+	int error;
+
+	value = nat64_get_output_method();
+	error = sysctl_handle_32(oidp, &value, 0, req);
+	/* Read operation or some error */
+	if ((error != 0) || (req->newptr == NULL))
+		return (error);
+	nat64_set_output_method(value);
+	return (0);
+}
+SYSCTL_PROC(_net_inet_ip_fw, OID_AUTO, nat64_direct_output,
+    CTLFLAG_VNET | CTLTYPE_U32 | CTLFLAG_RW, 0, 0, sysctl_direct_output, "IU",
+    "Use if_output directly instead of deffered netisr-based processing");
 
 static int
 vnet_ipfw_nat64_init(const void *arg __unused)

Modified: stable/11/sys/netpfil/ipfw/nat64/nat64_translate.c
==============================================================================
--- stable/11/sys/netpfil/ipfw/nat64/nat64_translate.c	Sun Apr 14 12:14:25 2019	(r346208)
+++ stable/11/sys/netpfil/ipfw/nat64/nat64_translate.c	Sun Apr 14 12:28:41 2019	(r346209)
@@ -25,8 +25,6 @@
  * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
-#include "opt_ipfw.h"
-
 #include <sys/cdefs.h>
 __FBSDID("$FreeBSD$");
 
@@ -71,6 +69,53 @@ __FBSDID("$FreeBSD$");
 #include "ip_fw_nat64.h"
 #include "nat64_translate.h"
 
+
+typedef int (*nat64_output_t)(struct ifnet *, struct mbuf *,
+    struct sockaddr *, struct nat64_counters *, void *);
+typedef int (*nat64_output_one_t)(struct mbuf *, struct nat64_counters *,
+    void *);
+
+static int nat64_find_route4(struct nhop4_basic *, struct sockaddr_in *,
+    struct mbuf *);
+static int nat64_find_route6(struct nhop6_basic *, struct sockaddr_in6 *,
+    struct mbuf *);
+static int nat64_output_one(struct mbuf *, struct nat64_counters *, void *);
+static int nat64_output(struct ifnet *, struct mbuf *, struct sockaddr *,
+    struct nat64_counters *, void *);
+static int nat64_direct_output_one(struct mbuf *, struct nat64_counters *,
+    void *);
+static int nat64_direct_output(struct ifnet *, struct mbuf *,
+    struct sockaddr *, struct nat64_counters *, void *);
+
+struct nat64_methods {
+	nat64_output_t		output;
+	nat64_output_one_t	output_one;
+};
+static const struct nat64_methods nat64_netisr = {
+	.output = nat64_output,
+	.output_one = nat64_output_one
+};
+static const struct nat64_methods nat64_direct = {
+	.output = nat64_direct_output,
+	.output_one = nat64_direct_output_one
+};
+static VNET_DEFINE(const struct nat64_methods *, nat64out) = &nat64_netisr;
+#define	V_nat64out	VNET(nat64out)
+
+void
+nat64_set_output_method(int direct)
+{
+
+	V_nat64out = direct != 0 ? &nat64_direct: &nat64_netisr;
+}
+
+int
+nat64_get_output_method(void)
+{
+
+	return (V_nat64out == &nat64_direct ? 1: 0);
+}
+
 static void
 nat64_log(struct pfloghdr *logdata, struct mbuf *m, sa_family_t family)
 {
@@ -80,14 +125,8 @@ nat64_log(struct pfloghdr *logdata, struct mbuf *m, sa
 	ipfw_bpf_mtap2(logdata, PFLOG_HDRLEN, m);
 }
 
-#ifdef IPFIREWALL_NAT64_DIRECT_OUTPUT
-static NAT64NOINLINE int nat64_find_route4(struct nhop4_basic *,
-    struct sockaddr_in *, struct mbuf *);
-static NAT64NOINLINE int nat64_find_route6(struct nhop6_basic *,
-    struct sockaddr_in6 *, struct mbuf *);
-
-static NAT64NOINLINE int
-nat64_output(struct ifnet *ifp, struct mbuf *m, struct sockaddr *dst,
+static int
+nat64_direct_output(struct ifnet *ifp, struct mbuf *m, struct sockaddr *dst,
     struct nat64_counters *stats, void *logdata)
 {
 	int error;
@@ -100,8 +139,9 @@ nat64_output(struct ifnet *ifp, struct mbuf *m, struct
 	return (error);
 }
 
-static NAT64NOINLINE int
-nat64_output_one(struct mbuf *m, struct nat64_counters *stats, void *logdata)
+static int
+nat64_direct_output_one(struct mbuf *m, struct nat64_counters *stats,
+    void *logdata)
 {
 	struct nhop6_basic nh6;
 	struct nhop4_basic nh4;
@@ -153,8 +193,8 @@ nat64_output_one(struct mbuf *m, struct nat64_counters
 		NAT64STAT_INC(stats, oerrors);
 	return (error);
 }
-#else /* !IPFIREWALL_NAT64_DIRECT_OUTPUT */
-static NAT64NOINLINE int
+
+static int
 nat64_output(struct ifnet *ifp, struct mbuf *m, struct sockaddr *dst,
     struct nat64_counters *stats, void *logdata)
 {
@@ -187,13 +227,12 @@ nat64_output(struct ifnet *ifp, struct mbuf *m, struct
 	return (ret);
 }
 
-static NAT64NOINLINE int
+static int
 nat64_output_one(struct mbuf *m, struct nat64_counters *stats, void *logdata)
 {
 
 	return (nat64_output(NULL, m, NULL, stats, logdata));
 }
-#endif /* !IPFIREWALL_NAT64_DIRECT_OUTPUT */
 
 /*
  * Check the given IPv6 prefix and length according to RFC6052:
@@ -426,12 +465,10 @@ nat64_init_ip4hdr(const struct ip6_hdr *ip6, const str
 	ip->ip_hl = sizeof(*ip) >> 2;
 	ip->ip_tos = (ntohl(ip6->ip6_flow) >> 20) & 0xff;
 	ip->ip_len = htons(sizeof(*ip) + plen);
-#ifdef IPFIREWALL_NAT64_DIRECT_OUTPUT
-	ip->ip_ttl = ip6->ip6_hlim - IPV6_HLIMDEC;
-#else
-	/* Forwarding code will decrement TTL. */
 	ip->ip_ttl = ip6->ip6_hlim;
-#endif
+	/* Forwarding code will decrement TTL for netisr based output. */
+	if (V_nat64out == &nat64_direct)
+		ip->ip_ttl -= IPV6_HLIMDEC;
 	ip->ip_sum = 0;
 	ip->ip_p = (proto == IPPROTO_ICMPV6) ? IPPROTO_ICMP: proto;
 	ip_fillid(ip);
@@ -649,7 +686,7 @@ nat64_icmp6_reflect(struct mbuf *m, uint8_t type, uint
 	icmp6->icmp6_cksum = in6_cksum(n, IPPROTO_ICMPV6,
 	    sizeof(struct ip6_hdr), plen);
 	m_freem(m);
-	nat64_output_one(n, stats, logdata);
+	V_nat64out->output_one(n, stats, logdata);
 	return;
 freeit:
 	NAT64STAT_INC(stats, dropped);
@@ -752,7 +789,7 @@ nat64_icmp_reflect(struct mbuf *m, uint8_t type,
 	icmp->icmp_cksum = in_cksum_skip(n, sizeof(struct ip) + plen,
 	    sizeof(struct ip));
 	m_freem(m);
-	nat64_output_one(n, stats, logdata);
+	V_nat64out->output_one(n, stats, logdata);
 	return;
 freeit:
 	NAT64STAT_INC(stats, dropped);
@@ -1169,12 +1206,10 @@ nat64_do_handle_ip4(struct mbuf *m, struct in6_addr *s
 
 	ip6.ip6_flow = htonl(ip->ip_tos << 20);
 	ip6.ip6_vfc |= IPV6_VERSION;
-#ifdef IPFIREWALL_NAT64_DIRECT_OUTPUT
-	ip6.ip6_hlim = ip->ip_ttl - IPTTLDEC;
-#else
-	/* Forwarding code will decrement HLIM. */
 	ip6.ip6_hlim = ip->ip_ttl;
-#endif
+	/* Forwarding code will decrement TTL for netisr based output. */
+	if (V_nat64out == &nat64_direct)
+		ip6.ip6_hlim -= IPTTLDEC;
 	ip6.ip6_plen = htons(plen);
 	ip6.ip6_nxt = (proto == IPPROTO_ICMP) ? IPPROTO_ICMPV6: proto;
 	/* Convert checksums. */
@@ -1207,7 +1242,7 @@ nat64_do_handle_ip4(struct mbuf *m, struct in6_addr *s
 	mbufq_init(&mq, 255);
 	nat64_fragment6(&cfg->stats, &ip6, &mq, m, nh.nh_mtu, ip_id, ip_off);
 	while ((m = mbufq_dequeue(&mq)) != NULL) {
-		if (nat64_output(nh.nh_ifp, m, (struct sockaddr *)&dst,
+		if (V_nat64out->output(nh.nh_ifp, m, (struct sockaddr *)&dst,
 		    &cfg->stats, logdata) != 0)
 			break;
 		NAT64STAT_INC(&cfg->stats, opcnt46);
@@ -1417,9 +1452,8 @@ nat64_handle_icmp6(struct mbuf *m, int hlen, uint32_t 
 	ip.ip_dst.s_addr = aaddr;
 	ip.ip_src.s_addr = nat64_extract_ip4(cfg, &ip6i->ip6_src);
 	/* XXX: Make fake ulp header */
-#ifdef IPFIREWALL_NAT64_DIRECT_OUTPUT
-	ip6i->ip6_hlim += IPV6_HLIMDEC; /* init_ip4hdr will decrement it */
-#endif
+	if (V_nat64out == &nat64_direct) /* init_ip4hdr will decrement it */
+		ip6i->ip6_hlim += IPV6_HLIMDEC;
 	nat64_init_ip4hdr(ip6i, ip6f, plen, proto, &ip);
 	m_adj(m, hlen - sizeof(struct ip));
 	bcopy(&ip, mtod(m, void *), sizeof(ip));
@@ -1589,7 +1623,7 @@ nat64_do_handle_ip6(struct mbuf *m, uint32_t aaddr, ui
 
 	m_adj(m, hlen - sizeof(ip));
 	bcopy(&ip, mtod(m, void *), sizeof(ip));
-	if (nat64_output(nh.nh_ifp, m, (struct sockaddr *)&dst,
+	if (V_nat64out->output(nh.nh_ifp, m, (struct sockaddr *)&dst,
 	    &cfg->stats, logdata) == 0)
 		NAT64STAT_INC(&cfg->stats, opcnt64);
 	return (NAT64RETURN);

Modified: stable/11/sys/netpfil/ipfw/nat64/nat64_translate.h
==============================================================================
--- stable/11/sys/netpfil/ipfw/nat64/nat64_translate.h	Sun Apr 14 12:14:25 2019	(r346208)
+++ stable/11/sys/netpfil/ipfw/nat64/nat64_translate.h	Sun Apr 14 12:28:41 2019	(r346209)
@@ -1,6 +1,6 @@
 /*-
- * Copyright (c) 2015-2016 Yandex LLC
- * Copyright (c) 2015-2016 Andrey V. Elsukov <ae@FreeBSD.org>
+ * Copyright (c) 2015-2018 Yandex LLC
+ * Copyright (c) 2015-2018 Andrey V. Elsukov <ae@FreeBSD.org>
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -141,6 +141,9 @@ void nat64_embed_ip4(const struct nat64_config *cfg, i
     struct in6_addr *ip6);
 in_addr_t nat64_extract_ip4(const struct nat64_config *cfg,
     const struct in6_addr *ip6);
+
+void nat64_set_output_method(int);
+int nat64_get_output_method(void);
 
 #endif