From owner-freebsd-pf@freebsd.org Mon Oct 15 22:26:56 2018 Return-Path: Delivered-To: freebsd-pf@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E65210CB017 for ; Mon, 15 Oct 2018 22:26:56 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D905D88B2E for ; Mon, 15 Oct 2018 22:26:55 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id EA270139B5 for ; Tue, 16 Oct 2018 00:26:45 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id JSQF4w1wIw2w for ; Tue, 16 Oct 2018 00:26:44 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id CD0601399F for ; Tue, 16 Oct 2018 00:26:44 +0200 (CEST) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id B497618F for ; Tue, 16 Oct 2018 00:26:44 +0200 (CEST) Message-ID: <5BC51424.5000309@incore.de> Date: Tue, 16 Oct 2018 00:26:44 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: freebsd-pf@freebsd.org Subject: rdr pass for proto tcp sometimes creates states with expire time zero and so breaking connections Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2018 22:26:56 -0000 On two of my FreeBSD 10 (r338093) firewall servers some incoming ssh connections stopped to work because pf started to create states with expire time zero (instead of 86400 sec) for translation statements like rdr pass on em0 proto tcp from any to myip port 8022 --> 10.0.0.254 Therefore a command given on a remote server like ssh -p 8022 myip sleep 15 did not return, because the created state for the connection was purged by pf (interval 10 sec) before 15 seconds. If I replace the rdr pass rule with a rdr rule and a separate filter pass rule then the created state always has expire time 24h and everything is ok. I have tried to analyze the bug in the rdr pass rule. Immediately after starting the above ssh command on the remote sever I see with pfctl -vss the sreated state on my firewall machine: all tcp 10.0.0.254:8022 (myip:8022) <- remoteip:59233 ESTABLISHED:ESTABLISHED [1443872812 + 66608] wscale 6 [1365307391 + 66560] wscale 3 age 00:00:00, expires in 00:00:00, 13:12 pkts, 2955:3306 bytes and a DTrace script running at the same time gives 3 19323 pfsync_state_export:entry creation=4491391. expire=4491391, state_timeout=2 3 16459 pf_state_expires:entry state_timeout=86400, start_timeout=6000, end_timeout=12000 states=882 3 17624 counter_u64_fetch:entry 3 17625 counter_u64_fetch:return returncode (states_cur)=4294967248 = 0xffffffd0 3 16460 pf_state_expires:return returncode=4491391 3 19324 pfsync_state_export:return creation=0. expire=0, syncstate_timeout=2 0 12730 pfioctl:return returncode=0, time_uptime=4491391 So pf_state_expires() returns the value time_update and therefore pfsync_state_export() gives expire zero. Reason for this is the very big (means negative) value returned by the function counter_u64_fetch() for states_cur. This variable is of type uint64_t and only incremented and decremented by the macros STATE_INC_COUNTERS and STATE_DEC_COUNTERS. I see two possibilities: Maybe counter(9) is broken for my hardware (i386, cpu_feature=0xbfebfbff) or STATE_DEC_COUNTERS sometimes is called without STATE_INC_COUNTERS called before. In the above example the value of states_cur is 1 on a fresh booted system, so we may have a difference of 49 in calls of the COUNTER macros. I see the same number 49 in the output of pfctl -si in the line for short packets. This may be a hint or its random. -- Andreas Longwitz