From owner-freebsd-pf@FreeBSD.ORG Fri Aug 15 14:58:19 2008 Return-Path: Delivered-To: freebsd-pf@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1EDF1065676 for ; Fri, 15 Aug 2008 14:58:19 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.186]) by mx1.freebsd.org (Postfix) with ESMTP id 240688FC1A for ; Fri, 15 Aug 2008 14:58:19 +0000 (UTC) (envelope-from max@love2party.net) Received: from vampire.homelinux.org (dslb-088-066-037-045.pools.arcor-ip.net [88.66.37.45]) by mrelayeu.kundenserver.de (node=mrelayeu6) with ESMTP (Nemesis) id 0ML29c-1KU0kr2WZQ-0001a9; Fri, 15 Aug 2008 16:58:18 +0200 Received: (qmail 51220 invoked from network); 15 Aug 2008 14:58:15 -0000 Received: from fbsd8.laiers.local (192.168.4.151) by mx.laiers.local with SMTP; 15 Aug 2008 14:58:15 -0000 From: Max Laier Organization: FreeBSD To: freebsd-pf@freebsd.org Date: Fri, 15 Aug 2008 16:58:15 +0200 User-Agent: KMail/1.10.0 (FreeBSD/8.0-CURRENT; KDE/4.1.0; i386; ; ) References: <8e10486b0808150708g200727b8sc2f4993eee9f5248@mail.gmail.com> In-Reply-To: <8e10486b0808150708g200727b8sc2f4993eee9f5248@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200808151658.15440.max@love2party.net> X-Provags-ID: V01U2FsdGVkX18MxRge4u3ja76yf7sv0LVRN9EZLSHk+sZU0C/ Jx0Hrwr6u2WtKI54JZ4E2vpYS60mFmyZMIQXfGiK9YeiW4Is6s +nWaf9OHSV7OfD/zQkQFg== Cc: Subject: Re: why BAD state messages X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Aug 2008 14:58:19 -0000 On Friday 15 August 2008 16:08:38 Alexandre Biancalana wrote: > Hi list, > > I'm experiencing some problems with blocked connections because of > bad states but I need some more information about why this is > happening, if this is timeout between tcp handshake, or state creation > or application trying to talk on closed connection. > > I have two FreeBSD 7-STABLE with PF, carp, pfsync and max carpdev > patch and two application servers (jboss) that listen on port 9090 > behind this firewalls, some connections from external clients off this > appservers are (apparently random) being blocked, enabling loud (pfctl > -x loud) I can see in /var/log/messages the following messages: > > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090 > 10.10.110.34:52347 [lo=3922530250 high=3922595445 win=65535 > modulator=0] [lo=3059100500 high=3059158735 win=65195 modulator=0] 4:4 > S seq=398900533 (398900533) ack=3059100500 len=0 ackskew=0 pkts=6:20 > dir=in,fwd > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090 > 10.10.110.34:50668 [lo=395881033 high=395946233 win=65535 modulator=0] > [lo=3568232053 high=3568290288 win=65200 modulator=0] 4:4 S > seq=2480335288 (2480335288) ack=3568232053 len=0 ackskew=0 pkts=6:20 > dir=in,fwd > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090 > 10.10.110.34:51582 [lo=3528357041 high=3528421509 win=65535 > modulator=0] [lo=3809540772 high=3809605893 win=64468 modulator=0] 9:9 > S seq=3810516558 (3810516558) ack=3809540772 len=0 ackskew=0 pkts=6:5 > dir=in,fwd > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090 > 10.10.110.34:50668 [lo=395881033 high=395946233 win=65535 modulator=0] > [lo=3568232053 high=3568290288 win=65200 modulator=0] 4:4 S > seq=2480335288 (2480335288) ack=3568232053 len=0 ackskew=0 pkts=6:20 > dir=in,fwd > kernel: pf: BAD state: TCP 10.10.6.18:9090 10.10.6.18:9090 > 10.10.81.242:2434 [lo=538716318 high=538780855 win=65535 modulator=0] > [lo=1004209856 high=1004274961 win=64537 modulator=0] 4:9 S > seq=1634723484 (1634723484) ack=1004209856 len=0 ackskew=0 pkts=5:4 > dir=in,fwd > > I tried to set custom tcp timeout options in this rules but this does not > help > > pass log proto tcp from any to { $apphpr01 $apphpr02 } port { 9090 } > keep state (tcp.opening 60, tcp.closed 180, tcp.finwait 90) > > > Any ideas on how can I know why this connections are being blocked ?? > I can provide any additional information needed. The blocked packets are SYNs. That means you are trying to reuse a port. This works if the state on both sides is >= FIN_WAIT2 (9) and you have pf.c r181291 (or one that has it merged). CVS rev 1.55 or 1.46.2.3 (RELENG_7) or apply the following patch: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/contrib/pf/net/pf.c.diff?r1=1.54;r2=1.55 This should fix the instances above where it says "...] 9:9 S ..." The others might be an artifact from pfsync or asymmetric routing? You can also mitigate the problem by giving your clients and the pf-forwarding a larger port range for outgoing connections. This is a typical problem if you open a large number of connections from one client (or load balancer) to one server. You can only have so many open at a given time. Check if you can enable streaming mode somehow. -- /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News