Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 12 Apr 2012 16:53:18 +0400
From:      "Dmitry S. Kasterin" <dmk.sbor@gmail.com>
To:        freebsd-ipfw@freebsd.org
Subject:   Stateful IPFW - too many connections in FIN_WAIT_2 or LAST_ACK states
Message-ID:  <CAJkxAbyMEYZ4pYu=z4Sfwdqtzh=PjhHE4qrbSsyL34YE9TnXZQ@mail.gmail.com>

next in thread | raw e-mail | index | archive | help
Hello!

I have rather simple ipfw ruleset like this:

00001 allow all from any to any via lo0

00010 check-state
00101 allow tcp from me to any out setup keep-state

65533 deny log ip from any to any
65534 deny ip6 from any to any

Actually, there are a few rules for upd, icmp and so on,
but the main idea here is to allow only outgoing (tcp) connections
and handle them using dynamic rules.

The first thing I have found was enormously high counter value on
"deny log ip from any to any" rule. For that moment my workstation was
placed in a small private (and "clean") network, so this value was
considered suspicious.
Later I've discovered that many tcp connections have FIN_WAIT_2 or LAST_ACK
state.

In order to determine what's going on I've carried out some
experiments (shown below).

Briefly: from my point of view, ipfw sometimes handles the last phase
of a connection improperly.
I was unable to reliably reproduce this behaviour - sometimes it
happens, but in the most cases not.
But when it happens, it leads to "frozen" connections.

Of course, this can be just a symptom of software misconfiguration or
maybe my mistake.
So I need an opinion from people with deep knoweledge of ipfw and
network stack.

Any information or comments are much appreciated!


PS I'm running 9.0-STABLE with custom kernel.


I.

1) Flush ipfw, reset counters, load fresh ruleset from file.

2) Run tcpdump on network interface (e.g. re0) and ipfw log interface (ipfw0)
# tcpdump -i re0 -p -w <from-medium.dump>
# tcpdump -i ipfw0 -p -w <from-ipfw.dump>

3) Disable proxy and make a query to a webserver, e.g.
$ lynx www.freebsd.org

4) Check ipfw counter and connections
# ipfw -deS show ; netstat -n -p tcp


In the most cases this test gives "normal" result. But under some circumstances
the result may be like this (w.x.y.z is an IP address of my workstation):

# ipfw -deS show ; netstat -n -p tcp
00001  0     0 set 0 allow ip from any to any via lo0
...
00010  0     0 set 0 check-state
00101 47 28622 set 0 allow tcp from me to any out setup keep-state
...
65533  6   312 set 0 deny log logamount 16 ip from any to any
## Dynamic rules (1):
00101 13  5620 (0s) STATE tcp w.x.y.z 26051 <-> 69.147.83.34 80

Active Internet connections
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 w.x.y.z.13414   69.147.83.34.80        LAST_ACK

So, the page (www.freebsd.org) was loaded and packets were counted.
But the dynamic rule entry is invalid - it has port 26051 instead of 13414.

The analysis of dump files has shown:

a) Dump from re0 has only one TCP stream:
w.x.y.z:13414 <-> 69.147.83.34:80

b) Dump from ipfw0:
N Time        Source         Destination           Protocol Length Info
1 0.000000    69.147.83.34   w.x.y.z               TCP      66
http > 13414 [ACK] Seq=1 Ack=1 Win=8325 Len=0 ...
2 0.557615    w.x.y.z        69.147.83.34          TCP      66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
3 1.947625    w.x.y.z        69.147.83.34          TCP      66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
4 4.527624    w.x.y.z        69.147.83.34          TCP      66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
5 9.487633    w.x.y.z        69.147.83.34          TCP      66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...
6 19.207616   w.x.y.z        69.147.83.34          TCP      66
13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ...

It looks like network stack tries to finalize connection but the
corresponding packets are dropped.


II.

1) Slightly change the ruleset:

00001   allow ip from any to any via lo0

00010   check-state
00101   allow tcp from me to any out setup keep-state

11001   allow log tcp from any 80 to me in
11002   allow log tcp from me to any dst-port 80 out

65533   deny ip from any to any
65534   deny ip6 from any to any

The idea is to see packets which were blocked in the previous test.

2) Flush ipfw, see I.
2) Run tcpdump, see I.
3) $ lynx www.freebsd.org
4) "ipfw -deS show" and "netstat -n -p tcp"

# ipfw -deS  show
00001   0     0 set 0 allow ip from any to any via lo0

00010   0     0 set 0 check-state
00101  33 22942 set 0 allow tcp from me to any out setup keep-state

11001   2    96 set 1 allow log logamount 16 tcp from any 80 to me in
11002   0     0 set 1 allow log logamount 16 tcp from me to any dst-port 80 out

65533   0     0 set 0 deny ip from any to any


An there are no waiting connections.

The dump from ipfw0 contains:
No.     Time        Source                Destination
Protocol Length Info
      1 0.000000    69.147.83.34          w.x.y.z        TCP      66
  http > 29470 [ACK] Seq=1 Ack=1 Win=8325 Len=0 ...


III.

1) Change ruleset back to:

00001 allow ip from any to any via lo0

00010 check-state
00101 allow tcp from me to any out setup keep-state

65534 deny ip6 from any to any
65535 deny ip from any to any

2) Browse the Internet.

3) I've visited some pages and now netstat output looks like:
# netstat -an -f inet | grep FIN_WAIT
tcp4       0      0 w.x.y.z.47536     173.194.32.21.443      FIN_WAIT_2
tcp4       0      0 w.x.y.z.47533     173.194.32.21.443      FIN_WAIT_2
tcp4       0      0 w.x.y.z.47532     173.194.32.21.443      FIN_WAIT_2
tcp4       0      0 w.x.y.z.47531     173.194.32.21.443      FIN_WAIT_2
tcp4       0      0 w.x.y.z.24851     199.7.55.72.80         FIN_WAIT_2
tcp4       0      0 w.x.y.z.24731     74.125.224.79.80       FIN_WAIT_2
tcp4       0      0 w.x.y.z.11578     213.180.204.69.80      FIN_WAIT_2
tcp4       0      0 w.x.y.z.11577     213.180.204.143.80     FIN_WAIT_2



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJkxAbyMEYZ4pYu=z4Sfwdqtzh=PjhHE4qrbSsyL34YE9TnXZQ>