Date: Thu, 12 Apr 2012 16:53:18 +0400 From: "Dmitry S. Kasterin" <dmk.sbor@gmail.com> To: freebsd-ipfw@freebsd.org Subject: Stateful IPFW - too many connections in FIN_WAIT_2 or LAST_ACK states Message-ID: <CAJkxAbyMEYZ4pYu=z4Sfwdqtzh=PjhHE4qrbSsyL34YE9TnXZQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello!
I have rather simple ipfw ruleset like this:
00001 allow all from any to any via lo0
00010 check-state
00101 allow tcp from me to any out setup keep-state
65533 deny log ip from any to any
65534 deny ip6 from any to any
Actually, there are a few rules for upd, icmp and so on,
but the main idea here is to allow only outgoing (tcp) connections
and handle them using dynamic rules.
The first thing I have found was enormously high counter value on
"deny log ip from any to any" rule. For that moment my workstation was
placed in a small private (and "clean") network, so this value was
considered suspicious.
Later I've discovered that many tcp connections have FIN_WAIT_2 or LAST_ACK
state.
In order to determine what's going on I've carried out some
experiments (shown below).
Briefly: from my point of view, ipfw sometimes handles the last phase
of a connection improperly.
I was unable to reliably reproduce this behaviour - sometimes it
happens, but in the most cases not.
But when it happens, it leads to "frozen" connections.
Of course, this can be just a symptom of software misconfiguration or
maybe my mistake.
So I need an opinion from people with deep knoweledge of ipfw and
network stack.
Any information or comments are much appreciated!
PS I'm running 9.0-STABLE with custom kernel.
I.
1) Flush ipfw, reset counters, load fresh ruleset from file.
2) Run tcpdump on network interface (e.g. re0) and ipfw log interface (ipfw0)
# tcpdump -i re0 -p -w <from-medium.dump>
# tcpdump -i ipfw0 -p -w <from-ipfw.dump>
3) Disable proxy and make a query to a webserver, e.g.
$ lynx www.freebsd.org
4) Check ipfw counter and connections
# ipfw -deS show ; netstat -n -p tcp
In the most cases this test gives "normal" result. But under some circumstances
the result may be like this (w.x.y.z is an IP address of my workstation):
# ipfw -deS show ; netstat -n -p tcp
00001 0 0 set 0 allow ip from any to any via lo0
...
00010 0 0 set 0 check-state
00101 47 28622 set 0 allow tcp from me to any out setup keep-state
...
65533 6 312 set 0 deny log logamount 16 ip from any to any
## Dynamic rules (1):
00101 13 5620 (0s) STATE tcp w.x.y.z 26051 <-> 69.147.83.34 80
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 w.x.y.z.13414 69.147.83.34.80 LAST_ACK
So, the page (www.freebsd.org) was loaded and packets were counted.
But the dynamic rule entry is invalid - it has port 26051 instead of 13414.
The analysis of dump files has shown:
a) Dump from re0 has only one TCP stream:
w.x.y.z:13414 <-> 69.147.83.34:80
b) Dump from ipfw0:
N Time Source Destination Protocol Length Info
1 0.000000 69.147.83.34 w.x.y.z TCP 66
http > 13414 [ACK] Seq=1 Ack=1 Win=8325 Len=0 ...
2 0.557615 w.x.y.z 69.147.83.34 TCP 66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
3 1.947625 w.x.y.z 69.147.83.34 TCP 66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
4 4.527624 w.x.y.z 69.147.83.34 TCP 66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
5 9.487633 w.x.y.z 69.147.83.34 TCP 66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
6 19.207616 w.x.y.z 69.147.83.34 TCP 66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
It looks like network stack tries to finalize connection but the
corresponding packets are dropped.
II.
1) Slightly change the ruleset:
00001 allow ip from any to any via lo0
00010 check-state
00101 allow tcp from me to any out setup keep-state
11001 allow log tcp from any 80 to me in
11002 allow log tcp from me to any dst-port 80 out
65533 deny ip from any to any
65534 deny ip6 from any to any
The idea is to see packets which were blocked in the previous test.
2) Flush ipfw, see I.
2) Run tcpdump, see I.
3) $ lynx www.freebsd.org
4) "ipfw -deS show" and "netstat -n -p tcp"
# ipfw -deS show
00001 0 0 set 0 allow ip from any to any via lo0
00010 0 0 set 0 check-state
00101 33 22942 set 0 allow tcp from me to any out setup keep-state
11001 2 96 set 1 allow log logamount 16 tcp from any 80 to me in
11002 0 0 set 1 allow log logamount 16 tcp from me to any dst-port 80 out
65533 0 0 set 0 deny ip from any to any
An there are no waiting connections.
The dump from ipfw0 contains:
No. Time Source Destination
Protocol Length Info
1 0.000000 69.147.83.34 w.x.y.z TCP 66
http > 29470 [ACK] Seq=1 Ack=1 Win=8325 Len=0 ...
III.
1) Change ruleset back to:
00001 allow ip from any to any via lo0
00010 check-state
00101 allow tcp from me to any out setup keep-state
65534 deny ip6 from any to any
65535 deny ip from any to any
2) Browse the Internet.
3) I've visited some pages and now netstat output looks like:
# netstat -an -f inet | grep FIN_WAIT
tcp4 0 0 w.x.y.z.47536 173.194.32.21.443 FIN_WAIT_2
tcp4 0 0 w.x.y.z.47533 173.194.32.21.443 FIN_WAIT_2
tcp4 0 0 w.x.y.z.47532 173.194.32.21.443 FIN_WAIT_2
tcp4 0 0 w.x.y.z.47531 173.194.32.21.443 FIN_WAIT_2
tcp4 0 0 w.x.y.z.24851 199.7.55.72.80 FIN_WAIT_2
tcp4 0 0 w.x.y.z.24731 74.125.224.79.80 FIN_WAIT_2
tcp4 0 0 w.x.y.z.11578 213.180.204.69.80 FIN_WAIT_2
tcp4 0 0 w.x.y.z.11577 213.180.204.143.80 FIN_WAIT_2
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJkxAbyMEYZ4pYu=z4Sfwdqtzh=PjhHE4qrbSsyL34YE9TnXZQ>