From owner-freebsd-net@FreeBSD.ORG Tue Apr 17 11:06:03 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 94BB8106564A for ; Tue, 17 Apr 2012 11:06:03 +0000 (UTC) (envelope-from dmk.sbor@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4E1D18FC17 for ; Tue, 17 Apr 2012 11:06:03 +0000 (UTC) Received: by yhgm50 with SMTP id m50so3516844yhg.13 for ; Tue, 17 Apr 2012 04:05:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=JEWUB2eODJBGgets7DQaJKwIufhvkmZLWrgwjCCjYTg=; b=WQ0/MWNlc/egk6qpHIGuA+uOa84M+UkNOxyNO0VqQGIPEQLWX8aq6tZP8bk7/UH+ou owu/2QK+bDgfHxBQ3lAP9Es9zP8wSoktkhJU33U1vaO9JETusaJORGQGb+RwCMJNoZLA efOcoMWbi2h1VHv3KsAQKkqZuwd2PujT/w+Nv3rwgy6/9v1mebHCT2wGMqMCqVy+NXnt yqZKEvtUqNHmj9xc/nrcTzKN+6wWePRmRqpxhaPSLekL+50HPY/MSUqUVbtulccCFaGk ynuEkPDbFosgpYPMDCkTJ5Gh85JajulNUDSXqQzXnABlvN7TBwOLQByDpFbfZxumO+bo tCAA== MIME-Version: 1.0 Received: by 10.236.185.138 with SMTP id u10mr14629893yhm.106.1334660757326; Tue, 17 Apr 2012 04:05:57 -0700 (PDT) Received: by 10.146.168.1 with HTTP; Tue, 17 Apr 2012 04:05:57 -0700 (PDT) In-Reply-To: References: Date: Tue, 17 Apr 2012 15:05:57 +0400 Message-ID: From: "Dmitry S. Kasterin" To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: Stateful IPFW - too many connections in FIN_WAIT_2 or LAST_ACK states X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Apr 2012 11:06:03 -0000 (Cross-posting this to net@ since there was no reply on ipfw@.) Hello! I have rather simple ipfw ruleset like this: 00001 allow all from any to any via lo0 00010 check-state 00101 allow tcp from me to any out setup keep-state 65533 deny log ip from any to any 65534 deny ip6 from any to any Actually, there are a few rules for upd, icmp and so on, but the main idea here is to allow only outgoing (tcp) connections and handle them using dynamic rules. The first thing I have found was enormously high counter value on "deny log ip from any to any" rule. For that moment my workstation was placed in a small private (and "clean") network, so this value was considered suspicious. Later I've discovered that many tcp connections have FIN_WAIT_2 or LAST_ACK state. In order to determine what's going on I've carried out some experiments (shown below). Briefly: from my point of view, ipfw sometimes handles the last phase of a connection improperly. I was unable to reliably reproduce this behaviour - sometimes it happens, but in the most cases not. But when it happens, it leads to "frozen" connections. Of course, this can be just a symptom of software misconfiguration or maybe my mistake. So I need an opinion from people with deep knoweledge of ipfw and network stack. Any information or comments are much appreciated! PS I'm running 9.0-STABLE with custom kernel. I. 1) Flush ipfw, reset counters, load fresh ruleset from file. 2) Run tcpdump on network interface (e.g. re0) and ipfw log interface (ipfw0) # tcpdump -i re0 -p -w # tcpdump -i ipfw0 -p -w 3) Disable proxy and make a query to a webserver, e.g. $ lynx www.freebsd.org 4) Check ipfw counter and connections # ipfw -deS show ; netstat -n -p tcp In the most cases this test gives "normal" result. But under some circumstances the result may be like this (w.x.y.z is an IP address of my workstation): # ipfw -deS show ; netstat -n -p tcp 00001 0 0 set 0 allow ip from any to any via lo0 ... 00010 0 0 set 0 check-state 00101 47 28622 set 0 allow tcp from me to any out setup keep-state ... 65533 6 312 set 0 deny log logamount 16 ip from any to any ## Dynamic rules (1): 00101 13 5620 (0s) STATE tcp w.x.y.z 26051 <-> 69.147.83.34 80 Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 w.x.y.z.13414 69.147.83.34.80 LAST_ACK So, the page (www.freebsd.org) was loaded and packets were counted. But the dynamic rule entry is invalid - it has port 26051 instead of 13414. The analysis of dump files has shown: a) Dump from re0 has only one TCP stream: w.x.y.z:13414 <-> 69.147.83.34:80 b) Dump from ipfw0: N Time Source Destination Protocol Length Info 1 0.000000 69.147.83.34 w.x.y.z TCP 66 http > 13414 [ACK] Seq=1 Ack=1 Win=8325 Len=0 ... 2 0.557615 w.x.y.z 69.147.83.34 TCP 66 13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ... 3 1.947625 w.x.y.z 69.147.83.34 TCP 66 13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ... 4 4.527624 w.x.y.z 69.147.83.34 TCP 66 13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ... 5 9.487633 w.x.y.z 69.147.83.34 TCP 66 13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ... 6 19.207616 w.x.y.z 69.147.83.34 TCP 66 13414 > http [FIN, ACK] Seq=0 Ack=1 Win=1040 Len=0 ... It looks like network stack tries to finalize connection but the corresponding packets are dropped. II. 1) Slightly change the ruleset: 00001 allow ip from any to any via lo0 00010 check-state 00101 allow tcp from me to any out setup keep-state 11001 allow log tcp from any 80 to me in 11002 allow log tcp from me to any dst-port 80 out 65533 deny ip from any to any 65534 deny ip6 from any to any The idea is to see packets which were blocked in the previous test. 2) Flush ipfw, see I. 2) Run tcpdump, see I. 3) $ lynx www.freebsd.org 4) "ipfw -deS show" and "netstat -n -p tcp" # ipfw -deS show 00001 0 0 set 0 allow ip from any to any via lo0 00010 0 0 set 0 check-state 00101 33 22942 set 0 allow tcp from me to any out setup keep-state 11001 2 96 set 1 allow log logamount 16 tcp from any 80 to me in 11002 0 0 set 1 allow log logamount 16 tcp from me to any dst-port 80 out 65533 0 0 set 0 deny ip from any to any An there are no waiting connections. The dump from ipfw0 contains: No. Time Source Destination Protocol Length Info 1 0.000000 69.147.83.34 w.x.y.z TCP 66 http > 29470 [ACK] Seq=1 Ack=1 Win=8325 Len=0 ... III. 1) Change ruleset back to: 00001 allow ip from any to any via lo0 00010 check-state 00101 allow tcp from me to any out setup keep-state 65534 deny ip6 from any to any 65535 deny ip from any to any 2) Browse the Internet. 3) I've visited some pages and now netstat output looks like: # netstat -an -f inet | grep FIN_WAIT tcp4 0 0 w.x.y.z.47536 173.194.32.21.443 FIN_WAIT_2 tcp4 0 0 w.x.y.z.47533 173.194.32.21.443 FIN_WAIT_2 tcp4 0 0 w.x.y.z.47532 173.194.32.21.443 FIN_WAIT_2 tcp4 0 0 w.x.y.z.47531 173.194.32.21.443 FIN_WAIT_2 tcp4 0 0 w.x.y.z.24851 199.7.55.72.80 FIN_WAIT_2 tcp4 0 0 w.x.y.z.24731 74.125.224.79.80 FIN_WAIT_2 tcp4 0 0 w.x.y.z.11578 213.180.204.69.80 FIN_WAIT_2 tcp4 0 0 w.x.y.z.11577 213.180.204.143.80 FIN_WAIT_2