From owner-freebsd-net@freebsd.org Wed Jun 13 17:16:13 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99778101F175; Wed, 13 Jun 2018 17:16:13 +0000 (UTC) (envelope-from jmk@wagsky.com) Received: from mx.allycomm.com (mx.allycomm.com [138.68.30.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A5FB6EA4E; Wed, 13 Jun 2018 17:16:12 +0000 (UTC) (envelope-from jmk@wagsky.com) Received: from JKLETSKY1-MBP15.local (184-23-190-180.vpn.dynamic.sonic.net [184.23.190.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx.allycomm.com (Postfix) with ESMTPSA id 5600428364; Wed, 13 Jun 2018 10:16:04 -0700 (PDT) From: Jeff Kletsky Subject: In-kernel NAT [ipfw] dropping large UDP return packets To: freebsd-net@freebsd.org, freebsd-ipfw@freebsd.org Message-ID: Date: Wed, 13 Jun 2018 10:16:04 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jun 2018 17:16:14 -0000 When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC tunnel to the T-Mobile provisioning servers, the reassembled, 4640-byte return packet is silently dropped by the in-kernel NAT, even though it "matches" the outbound packet from less than 100 ms prior. All other operations of the firewall seem to be functioning as expected. This includes iPhones using "WiFi Calling" which utilizes similar IPSEC connections to T-Mobile servers (though fragmentation has not been seen on those connections). The connection for the femto-cell can be handled by a Linux/netfilter NAT. Proper reassembly of the packet fragments within the firewall ("reass ip from any to any"), at the start of the rule set, prior to NAT, has been confirmed with ngtee and wireshark. Is there a known issue with large packets and in-kernel NAT? The only sysctl that I found that seemed related was the UDP timeout. For good measure I upped it to 30 (seconds), but that did not change the behavior. Are there known causes and/or resolutions for this behavior? Is there a way to be able to "monitor" the NAT table? (I didn't see anything obvious in the ipfw, natd, or libalias man pages.) Thanks! Jeff 11.1-RELEASE-p9 FreeBSD 11.1-RELEASE-p9 #0: Tue Apr  3 16:59:16 UTC 2018 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 Additional details at https://forums.freebsd.org/threads/in-kernel-nat-dropping-large-udp-return-packets.66262/ pcapng files and ipfw rule set available on request to freebsd {a} wagsky {.} com