Date: Fri, 15 Aug 2008 13:26:31 -0300 From: "Alexandre Biancalana" <biancalana@gmail.com> To: "Max Laier" <max@love2party.net> Cc: freebsd-pf@freebsd.org Subject: Re: why BAD state messages Message-ID: <8e10486b0808150926m7e25bcedw34b24c2e7707e445@mail.gmail.com> In-Reply-To: <200808151658.15440.max@love2party.net> References: <8e10486b0808150708g200727b8sc2f4993eee9f5248@mail.gmail.com> <200808151658.15440.max@love2party.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8/15/08, Max Laier <max@love2party.net> wrote:
> On Friday 15 August 2008 16:08:38 Alexandre Biancalana wrote:
> > Hi list,
> >
> > I'm experiencing some problems with blocked connections because of
> > bad states but I need some more information about why this is
> > happening, if this is timeout between tcp handshake, or state creation
> > or application trying to talk on closed connection.
> >
> > I have two FreeBSD 7-STABLE with PF, carp, pfsync and max carpdev
> > patch and two application servers (jboss) that listen on port 9090
> > behind this firewalls, some connections from external clients off this
> > appservers are (apparently random) being blocked, enabling loud (pfctl
> > -x loud) I can see in /var/log/messages the following messages:
> >
> > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
> > 10.10.110.34:52347 [lo=3922530250 high=3922595445 win=65535
> > modulator=0] [lo=3059100500 high=3059158735 win=65195 modulator=0] 4:4
> > S seq=398900533 (398900533) ack=3059100500 len=0 ackskew=0 pkts=6:20
> > dir=in,fwd
> > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
> > 10.10.110.34:50668 [lo=395881033 high=395946233 win=65535 modulator=0]
> > [lo=3568232053 high=3568290288 win=65200 modulator=0] 4:4 S
> > seq=2480335288 (2480335288) ack=3568232053 len=0 ackskew=0 pkts=6:20
> > dir=in,fwd
> > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
> > 10.10.110.34:51582 [lo=3528357041 high=3528421509 win=65535
> > modulator=0] [lo=3809540772 high=3809605893 win=64468 modulator=0] 9:9
> > S seq=3810516558 (3810516558) ack=3809540772 len=0 ackskew=0 pkts=6:5
> > dir=in,fwd
> > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
> > 10.10.110.34:50668 [lo=395881033 high=395946233 win=65535 modulator=0]
> > [lo=3568232053 high=3568290288 win=65200 modulator=0] 4:4 S
> > seq=2480335288 (2480335288) ack=3568232053 len=0 ackskew=0 pkts=6:20
> > dir=in,fwd
> > kernel: pf: BAD state: TCP 10.10.6.18:9090 10.10.6.18:9090
> > 10.10.81.242:2434 [lo=538716318 high=538780855 win=65535 modulator=0]
> > [lo=1004209856 high=1004274961 win=64537 modulator=0] 4:9 S
> > seq=1634723484 (1634723484) ack=1004209856 len=0 ackskew=0 pkts=5:4
> > dir=in,fwd
> >
> > I tried to set custom tcp timeout options in this rules but this does not
> > help
> >
> > pass log proto tcp from any to { $apphpr01 $apphpr02 } port { 9090 }
> > keep state (tcp.opening 60, tcp.closed 180, tcp.finwait 90)
> >
> >
> > Any ideas on how can I know why this connections are being blocked ??
> > I can provide any additional information needed.
>
>
> The blocked packets are SYNs. That means you are trying to reuse a port.
> This works if the state on both sides is >= FIN_WAIT2 (9) and you have pf.c
> r181291 (or one that has it merged). CVS rev 1.55 or 1.46.2.3 (RELENG_7) or
> apply the following patch:
> http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/contrib/pf/net/pf.c.diff?r1=1.54;r2=1.55
Hi Max ! Thank you for your quick the reply.
Looking at machine's pf.c this looks too old
# grep FBSD /usr/src/sys/contrib/pf/net/pf.c
__FBSDID("$FreeBSD: src/sys/contrib/pf/net/pf.c,v 1.46.2.1 2007/11/25
19:26:46 mlaier Exp $");
I will do a csup, rebuild the kernel and see how this improve the situation.
>
> This should fix the instances above where it says "...] 9:9 S ..." The others
> might be an artifact from pfsync or asymmetric routing? You can also mitigate
> the problem by giving your clients and the pf-forwarding a larger port range
> for outgoing connections. This is a typical problem if you open a large
> number of connections from one client (or load balancer) to one server. You
> can only have so many open at a given time. Check if you can enable streaming
> mode somehow.
Looking the logs I made some math on each state
9:9 6174 times
4:4 3283 times
4:9 2611 times
10:10 1382 times
2:0 878 times
9:4 520 times
How can I give a larger range for outgoing conections if the clients
connect directly to the servers ? In this case I don't have any rdr
rule.
Thank you again!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8e10486b0808150926m7e25bcedw34b24c2e7707e445>