From owner-freebsd-net@freebsd.org Fri Feb 22 16:09:57 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4071114F1C73 for ; Fri, 22 Feb 2019 16:09:57 +0000 (UTC) (envelope-from olgeni@olgeni.com) Received: from hub.olgeni.com (hub.olgeni.com [31.171.246.156]) by mx1.freebsd.org (Postfix) with ESMTP id 481A5735D0 for ; Fri, 22 Feb 2019 16:09:55 +0000 (UTC) (envelope-from olgeni@olgeni.com) Received: from [192.168.0.80] (unknown [82.84.91.88]) by hub.olgeni.com (Postfix) with ESMTPSA id 3A90FD7916 for ; Fri, 22 Feb 2019 17:09:48 +0100 (CET) Date: Fri, 22 Feb 2019 17:09:46 +0100 (CET) From: Giacomo Olgeni X-X-Sender: olgeni@macbook.local To: freebsd-net@FreeBSD.org Subject: IPFW NAT in VNET jail Message-ID: User-Agent: Alpine 2.21 (OSX 202 2017-01-01) X-OpenPGP-KeyID: 0x90B7A98E6450AE47 X-OpenPGP-Fingerprint: 7133 AB4D DFC8 0A0D F891 B0D2 90B7 A98E 6450 AE47 X-OpenPGP-URL: http://hub.olgeni.com/~olgeni/pgp/olgeni@olgeni.com MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 481A5735D0 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of olgeni@olgeni.com designates 31.171.246.156 as permitted sender) smtp.mailfrom=olgeni@olgeni.com X-Spamd-Result: default: False [-1.62 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.77)[-0.767,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.96)[-0.964,0]; DMARC_NA(0.00)[olgeni.com]; NEURAL_SPAM_SHORT(0.31)[0.315,0]; MX_GOOD(-0.01)[hub.olgeni.com]; IP_SCORE(0.01)[country: CH(0.03)]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:50837, ipnet:31.171.244.0/22, country:CH]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Feb 2019 16:09:57 -0000 Hi, I have been playing around with IPFW and in-kernel NAT in a VNET jail, in order to implement NAT before IPsec. The IPsec part is provided by a different VNET jail that seems to be working fine. Problem is, I could not get NAT working in any way so far. And just to be sure, I already disabled anything related to checksum offloading from all the physical network cards (txcsum, rxcsum, tso, etc.). My routing is from epair1b (192.168.3.0/24) to epair0b (10.64.23.34, using 10.64.23.33 as gw). I can see that NAT is kind of doing what is expected, but then something weird happens. I run "telnet 10.101.214.155 80" (to reach the remote IPsec subnet) from 192.168.3.9; connection goes through 192.168.3.223 (the gateway jail, never mind my temporary addresses), and from there it goes out through NAT to the IPsec jail. tcpdump on the NAT jail (outbound interface epair0b) shows this: 17:00:04.330364 IP (tos 0x10, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.64.23.34.56647 > 10.101.214.155.http: Flags [S], cksum 0x4112 (correct), seq 3633527062, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1310639916 ecr 0], length 0 17:00:04.349254 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto TCP (6), length 56) 10.101.214.155.http > 10.64.23.34.56647: Flags [S.], cksum 0xede4 (correct), seq 619471197, ack 3633527063, win 4344, options [mss 1360,sackOK,TS val 91081438 ecr 1310639916], length 0 17:00:04.349464 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40) 10.64.23.34.58666 > 10.101.214.155.http: Flags [R], cksum 0xb658 (correct), seq 3633527063, win 0, length 0 So, I'm get the right SYN/ACK to the right port, but then a RST is sent from an unrelated port, and the connection starts retransmitting the SYN. I have "nat 1 config log if epair0b", and "ipfw show" lists the following rules: 00100 0 0 nat 1 ip from any to any recv epair0b 00200 74 4080 nat 1 ip from any to any xmit epair0b 00300 0 0 check-state :default 00400 6 360 allow tcp from any to any out xmit epair0b setup keep-state :default 00500 0 0 allow udp from any to any out xmit epair0b setup 65535 113149 11125118 allow ip from any to any I'm running with one_pass=1. If I set one_pass=0 the only change is that rule 400 is hit by packets going out from NAT, but the result doesn't change. I added rule 100 to see if I had any match from the incoming SYN/ACK, but the counter is always 0. Rule 300 has 0 packets too (even with one_pass=0), so it's not clear to me whether the SYN/ACK is actually being seen by the firewall or if the RST is coming from somewhere else. I tried to run dwatch from the jail host and got this: # sudo dwatch -X tcp | grep 10.64.23.34 INFO Sourcing tcp profile [found in /usr/libexec/dwatch] INFO Watching 'tcp:::accept-established, [...]' 2019 Feb 22 17:00:04 0.0 intr[12]: 10.64.23.34:56647 <- 10.101.214.155:80 34 bytes 2019 Feb 22 17:00:04 0.0 intr[12]: 10.64.23.34:56647 X- 10.101.214.155:80 2019 Feb 22 17:00:04 0.0 intr[12]: 10.64.23.34:56647 -> 10.101.214.155:80 19 bytes The events look the same but there's no port 58666 anywhere here, which is where the RST comes from. My naive interpretation would be that NAT has nowhere to map the SYN/ACK for some reason, possibly because rule 100 is never hit? But that wouldn't explain the odd port number. I'm a bit out of ideas short of not using a VNET jail and trying to do everything from a bhyve VM, so any suggestion/interpretation is welcome :)