Date: Mon, 3 Oct 2005 12:05:12 GMT From: Vladimir Kotal <vlada@devnull.cz> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/86871: allocation logic for PCBs in TIME_WAIT state causes packet drops on stateful FWs Message-ID: <200510031205.j93C5CAZ054115@www.freebsd.org> Resent-Message-ID: <200510031210.j93CABqf082378@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 86871
>Category: kern
>Synopsis: allocation logic for PCBs in TIME_WAIT state causes packet drops on stateful FWs
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Oct 03 12:10:10 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator: Vladimir Kotal
>Release: 5.4-RELEASE-p7
>Organization:
>Environment:
FreeBSD ipfw2 5.4-RELEASE-p7 FreeBSD 5.4-RELEASE-p7 #7: Mon Oct 3 11:48:16 CEST 2005 lada@ipfw2:/sys/i386/compile/GENERIC i386
>Description:
From FreeBSD 5.x on, it is possible to allocate local ports for TCP connections even if PCB for given socket is in TIME_WAIT state. This causes execessive packet drops in scenarios where client initiates lots of outgoing connections toward server with stateful firewall (pf).
This behavior is not present in 4.x releases.
>How-To-Repeat:
1. download and compile
http://techie.devnull.cz/public/tcpclient-bind.c
[ or use any other basic TCP client which can bind(2) local port ]
2. connect to TCP service running on server with stateful firewall.
3. wait for tcpclient-bind to end
4. display socket state for the connection from step nr. 2
5. connect to the same TCP service
6. watch server logs (dmesg) for 'pf BAD state' entries
For example:
lada@ipfw2$ ./tcpclient-bind SERVER 21
connecting
connected
src addr = CLIENT-IP
src port = 1234
closing connection
lada@ipfw2$ netstat -an |grep SERVER-IP
tcp4 0 0 CLIENT-IP.1234 SERVER-IP.21 TIME_WAIT
lada@ipfw2$ ./tcpclient-bind SERVER 21
connecting
tcpclient-bind: connect: Operation timed out
lada@ipfw2$
the 'BAD state' messages then look like this:
Oct 3 13:53:54 SERVER kernel: pf: BAD state: TCP SERVER-IP:21 SERVER-IP:21 CLIENT-IP:4094 [lo=1074578133 high=1074644741 win=33304 modulator=0 wscale=1] [lo=819316103 high=819382710 win=33304 modulator=0 wscale=1] 9:9 S seq=1087912989 ack=819316103 len=0 ackskew=0 pkts=12:19 dir=in,fwd
Oct 3 13:53:54 SERVER kernel: pf: State failure on: 1 | 5
example of stateful rule:
pass in quick on $ext_if proto tcp from any to ($ext_if) \
port = ftp \
flags S/S keep state \
label "EXT-ingress--ftp-cmd"
This problem is always reproducible.
One of the scenarios where this problems could manifest itself is e.g. proxy+backend setup where proxy issues big number of connections to backend server running stateful firewall.
>Fix:
There is number of possibilities:
- convert stateful firewall ruleset to stateless (really not an option in my setup)
- tweak timeouts for firewall states (also not an option)
- remove INP_TIMEWAIT logic from FreeBSD 5.x TCP stack altogether
- use following patch:
http://techie.devnull.cz/public/noTW.patch
- this patch adds new sysctl net.inet.ip.portrange.twalloc which can be used to revert the INP_TIMEWAIT behavior to the old state. the patch will preserve current bind(2)/connect(2) behavior in FreeBSD 5.x by default.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200510031205.j93C5CAZ054115>