From owner-freebsd-pf@FreeBSD.ORG Wed Dec 19 07:18:23 2007 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41BED16A418 for ; Wed, 19 Dec 2007 07:18:23 +0000 (UTC) (envelope-from silver.salonen@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.157]) by mx1.freebsd.org (Postfix) with ESMTP id B269E13C45B for ; Wed, 19 Dec 2007 07:18:22 +0000 (UTC) (envelope-from silver.salonen@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so536085fgg.35 for ; Tue, 18 Dec 2007 23:18:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:message-id; bh=xeQSEisIClGw58E6U5zzz2QyTyzFeXHOBRAwDIxFjWc=; b=FX3O08FNO17WdwWM5XMAyXP8DZEYa2T6xp6RNTjYmYnyiR8m4R0dD9e+Bymz3zPumAUD297XvTVXNDlP4v6BBnJ1Zwe59Td9LGRqipVeYeH2WfTXaUYNLc8uVXFG9+/TTkogu56RnJzt5LBY6tol5400ycBxkxcE9NmG42Hyjt4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:message-id; b=i1qCiFswzD5Q597AiGSma/Z5QB/DeziLuWzV/Mp2dKGqFV4Pa3NldqIqx/hMMrtSCViPOsiAg8HFAx8o9phhBcywCdgML2/AFc++ey+lG5clQG4di1MIjAhF3cGU77+NndswY+R6TgY/aX8MA+NWZ5xwfOfmOoaQHt6h4uqSXo8= Received: by 10.86.73.17 with SMTP id v17mr8574198fga.74.1198048701545; Tue, 18 Dec 2007 23:18:21 -0800 (PST) Received: from ?192.168.8.99? ( [195.50.198.178]) by mx.google.com with ESMTPS id e32sm2426605fke.2007.12.18.23.18.19 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 18 Dec 2007 23:18:20 -0800 (PST) From: Silver Salonen To: freebsd-pf@freebsd.org Date: Wed, 19 Dec 2007 09:18:15 +0200 User-Agent: KMail/1.9.7 References: <200712180934.58755.silver.salonen@gmail.com> <14397207.post@talk.nabble.com> <200712181828.04998.max@love2party.net> In-Reply-To: <200712181828.04998.max@love2party.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200712190918.16730.silver.salonen@gmail.com> Cc: Subject: Re: occasional "Operation not permitted" on state-mismatch X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2007 07:18:23 -0000 On Tuesday 18 December 2007 19:27, Max Laier wrote: > On Tuesday 18 December 2007, Atrox wrote: > > Atrox wrote: > > > Hello! > > > > > > I have some FreeBSD-boxes (2x6.3-PRERELEASE (installed on 08.Dec), > > > 1x6.2-RELEASE) with PF configured. They are connected with OpenVPN > > > LAN-to-LAN > > > and the problem is that a few times per hour connection drops between > > > computers from one LAN to another. At first I blamed OpenVPN, then I > > > blamed > > > bridge, but now I've realized that the problem is in PF. > > > So I've tried increasing TCP-timeouts and setting optimization > > > to "aggressive", but well, it's still the same. > > > > > > I monitor connections by sending TCP packets once per second to some > > > other host and wait for reply. I use Nagios-plugins' check_tcp for > > > that. The script > > > looks like: > > > ===== > > > while [ 1 ]; do > > > pfctl -si |grep mismatch > > > /usr/local/libexec/nagios/check_tcp -H $host -p $port -t 2 > > > pfctl -si |grep mismatch > > > sleep 1 > > > done > > > ===== > > > > > > So if I let this script into action, I see that in 2-3 minutes, > > > check_tcp gets "Operation not permitted" error and just in this > > > moment packet-mismatch > > > counter is increased by one (on machine with lesser traffic, I get > > > the timeout > > > in a few hours). That's on both 6.3-PRERELEASE as well as on > > > 6.2-RELEASE. I've > > > tried connections: > > > * along WAN to IPFW-enabled machines > > > * along WAN to PF-enabled machines > > > * along LAN to PF-enabled machines > > > * along LAN to Windows machines > > > * along VPN to PF-enabled machines > > > * along VPN to Windows machines > > > > > > Sometimes I get just some connection timeout: CRITICAL - Socket > > > timeout after > > > 2 seconds (I don't know what could cause that). > > > > > > I can see this behaviour in about every FreeBSD/PF machine I have. > > > > > > The basic PF-configuration looks like: > > > ===== > > > set block-policy return > > > set loginterface $ext_if > > > set timeout tcp.closed 15 > > > set optimization aggressive > > > scrub in all no-df > > > > > > block drop out quick on $ext_if from ($ext_if) to 0.0.0.0 > > > block log all > > > pass quick on lo0 all > > > pass out all modulate state > > > pass out proto tcp all flags S/SA modulate state > > > pass on $int_if all modulate state > > > pass on $int_if proto tcp all flags S/SA modulate state > > > pass in on $ext_if proto tcp from any to ($ext_if) port $tcp_services > > > flags > > > S/SA modulate state > > > ===== > > > > > > Is PF buggy or have I misconfigured smth? > > > > Today I installed an OpenBSD-4.2 box just to see whether PF does the > > same thing there. And yes, it does. > > pf.conf: > > ===== > > ext_if = rl0 > > set block-policy return > > set loginterface $ext_if > > scrub in all no-df > > block drop out quick on $ext_if from ($ext_if) to 0.0.0.0 > > pass all modulate state > > pass quick on lo0 all > > ===== > > > > I check TCP without "sleep 1" now, and I do it to FreeBSD box without > > firewall. state-mismatch gets increased by one, and I get either "No > > route to host" or "Socket timeout after 2 seconds". > > > > Am I still misconfiguring the thing? > > No idea, you don't give nearly enough data for us to figure out what your > setup looks like. Well, as for OpenBSD, I installed the default system and changed only the pf.conf, and it sits in LAN (.../24). It's quite plain actually. > Regular tcp state mismatch usually hints that pf isn't > seeing all packets of the conversation. This can be caused by triangular > routing, load balanceing or if_bridge (which is difficult to get right in > some scenarios). Does using if_bridge matter when I test connections on non-bridged interfaces? I see state-mismatch while testing TCP-connection on servers' external interfaces (along WAN) and these are not in bridge. -- Silver