From owner-freebsd-net@freebsd.org Thu Feb 11 15:08:48 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 356F752AEC3 for ; Thu, 11 Feb 2021 15:08:48 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4Dc0Q814tXz3slf for ; Thu, 11 Feb 2021 15:08:48 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 250E152AEC2; Thu, 11 Feb 2021 15:08:48 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 24D5552AA4A for ; Thu, 11 Feb 2021 15:08:48 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dc0Q74WxGz3sj9; Thu, 11 Feb 2021 15:08:47 +0000 (UTC) (envelope-from avg@FreeBSD.org) X-Originating-IP: 195.64.148.76 Received: from [192.168.0.88] (unknown [195.64.148.76]) (Authenticated sender: andriy.gapon@uabsd.com) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 0E51B24001B; Thu, 11 Feb 2021 15:08:43 +0000 (UTC) To: net@FreeBSD.org Cc: "Andrey V. Elsukov" From: Andriy Gapon Subject: ipfw stateful rules and quick port re-use Message-ID: <5ccab312-085c-b764-97c9-4c2bc846cd22@FreeBSD.org> Date: Thu, 11 Feb 2021 17:08:43 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.7.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4Dc0Q74WxGz3sj9 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; ASN(0.00)[asn:29169, ipnet:217.70.176.0/20, country:FR] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Feb 2021 15:08:48 -0000 Recently we encountered an interesting issue at work. By accident our software started to quickly re-use a source TCP port when connecting to a remote system. That is, after a graceful shutdown of a connection (two FINs, etc), the software would quickly establish an identical connection by re-using the same local port and connecting to the same remote end-point. That did not work well for the application :) We saw problems where packets from the second connection would be dropped by ipfw. That happened because there would be no dynamic rule to let the packets through even though the first connection worked without any issues. >From a quick glance at the code it seems that the TCP protocol state kept by ipfw for dynamic rules is "append-only". That is, bits can be set in it but never cleared. So, when the first connection is closed the dynamic has "both syn" and "both fin" bits. When the second connection is established before the rule is expired, the rule is re-used for it, but its state remains the same. And its expiry time remains dyn_fin_lifetime. I think that that opens a race between the expiry timer (running every second) and the connection's packets given the short lifetime. Maybe I misanalyzed the situation and it's probably very rare. But still it's a valid use of TCP, so maybe ipfw could support it better (e.g., by detecting "syn" after "both fin"). -- Andriy Gapon