From owner-freebsd-bugs@FreeBSD.ORG Mon Oct 3 12:10:19 2005 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 31E8016A420 for ; Mon, 3 Oct 2005 12:10:19 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3A65043D55 for ; Mon, 3 Oct 2005 12:10:11 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.3/8.13.3) with ESMTP id j93CAB8L082388 for ; Mon, 3 Oct 2005 12:10:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.3/8.13.1/Submit) id j93CABqf082378; Mon, 3 Oct 2005 12:10:11 GMT (envelope-from gnats) Resent-Date: Mon, 3 Oct 2005 12:10:11 GMT Resent-Message-Id: <200510031210.j93CABqf082378@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Vladimir Kotal Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C6DE16A41F for ; Mon, 3 Oct 2005 12:05:13 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [216.136.204.117]) by mx1.FreeBSD.org (Postfix) with ESMTP id D17EA43D48 for ; Mon, 3 Oct 2005 12:05:12 +0000 (GMT) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.13.1/8.13.1) with ESMTP id j93C5CZg054116 for ; Mon, 3 Oct 2005 12:05:12 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.13.1/8.13.1/Submit) id j93C5CAZ054115; Mon, 3 Oct 2005 12:05:12 GMT (envelope-from nobody) Message-Id: <200510031205.j93C5CAZ054115@www.freebsd.org> Date: Mon, 3 Oct 2005 12:05:12 GMT From: Vladimir Kotal To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-2.3 Cc: Subject: kern/86871: allocation logic for PCBs in TIME_WAIT state causes packet drops on stateful FWs X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Oct 2005 12:10:19 -0000 >Number: 86871 >Category: kern >Synopsis: allocation logic for PCBs in TIME_WAIT state causes packet drops on stateful FWs >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Oct 03 12:10:10 GMT 2005 >Closed-Date: >Last-Modified: >Originator: Vladimir Kotal >Release: 5.4-RELEASE-p7 >Organization: >Environment: FreeBSD ipfw2 5.4-RELEASE-p7 FreeBSD 5.4-RELEASE-p7 #7: Mon Oct 3 11:48:16 CEST 2005 lada@ipfw2:/sys/i386/compile/GENERIC i386 >Description: From FreeBSD 5.x on, it is possible to allocate local ports for TCP connections even if PCB for given socket is in TIME_WAIT state. This causes execessive packet drops in scenarios where client initiates lots of outgoing connections toward server with stateful firewall (pf). This behavior is not present in 4.x releases. >How-To-Repeat: 1. download and compile http://techie.devnull.cz/public/tcpclient-bind.c [ or use any other basic TCP client which can bind(2) local port ] 2. connect to TCP service running on server with stateful firewall. 3. wait for tcpclient-bind to end 4. display socket state for the connection from step nr. 2 5. connect to the same TCP service 6. watch server logs (dmesg) for 'pf BAD state' entries For example: lada@ipfw2$ ./tcpclient-bind SERVER 21 connecting connected src addr = CLIENT-IP src port = 1234 closing connection lada@ipfw2$ netstat -an |grep SERVER-IP tcp4 0 0 CLIENT-IP.1234 SERVER-IP.21 TIME_WAIT lada@ipfw2$ ./tcpclient-bind SERVER 21 connecting tcpclient-bind: connect: Operation timed out lada@ipfw2$ the 'BAD state' messages then look like this: Oct 3 13:53:54 SERVER kernel: pf: BAD state: TCP SERVER-IP:21 SERVER-IP:21 CLIENT-IP:4094 [lo=1074578133 high=1074644741 win=33304 modulator=0 wscale=1] [lo=819316103 high=819382710 win=33304 modulator=0 wscale=1] 9:9 S seq=1087912989 ack=819316103 len=0 ackskew=0 pkts=12:19 dir=in,fwd Oct 3 13:53:54 SERVER kernel: pf: State failure on: 1 | 5 example of stateful rule: pass in quick on $ext_if proto tcp from any to ($ext_if) \ port = ftp \ flags S/S keep state \ label "EXT-ingress--ftp-cmd" This problem is always reproducible. One of the scenarios where this problems could manifest itself is e.g. proxy+backend setup where proxy issues big number of connections to backend server running stateful firewall. >Fix: There is number of possibilities: - convert stateful firewall ruleset to stateless (really not an option in my setup) - tweak timeouts for firewall states (also not an option) - remove INP_TIMEWAIT logic from FreeBSD 5.x TCP stack altogether - use following patch: http://techie.devnull.cz/public/noTW.patch - this patch adds new sysctl net.inet.ip.portrange.twalloc which can be used to revert the INP_TIMEWAIT behavior to the old state. the patch will preserve current bind(2)/connect(2) behavior in FreeBSD 5.x by default. >Release-Note: >Audit-Trail: >Unformatted: