From owner-freebsd-net@FreeBSD.ORG Fri Jul 18 14:32:53 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 864371065675 for ; Fri, 18 Jul 2008 14:32:53 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id C89678FC12 for ; Fri, 18 Jul 2008 14:32:52 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id m6IDxaao029101 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Fri, 18 Jul 2008 15:59:37 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by cicely5.cicely.de (8.14.2/8.14.2) with ESMTP id m6IDxWbi026162 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 18 Jul 2008 15:59:32 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id m6IDxW51048121; Fri, 18 Jul 2008 15:59:32 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id m6IDxWjT048120; Fri, 18 Jul 2008 15:59:32 +0200 (CEST) (envelope-from ticso) Date: Fri, 18 Jul 2008 15:59:31 +0200 From: Bernd Walter To: freebsd-net@freebsd.org Message-ID: <20080718135931.GA48087@cicely7.cicely.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-4.3 required=5.0 tests=ALL_TRUSTED=-1.8, AWL=0.146, BAYES_00=-2.599 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on spamd.cicely.de Cc: Bernd Walter Subject: TCP zombie connections with 7-RELEASE and STABLE from 15th june X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jul 2008 14:32:53 -0000 14:45:58.109631 IP 213.83.6.106.3270 > 85.159.14.110.443: S 470580731:470580731(0) win 32768 14:45:58.109753 IP 85.159.14.110.443 > 213.83.6.106.3270: S 1364510055:1364510055(0) ack 470580732 win 65535 14:45:58.114324 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 1 win 33304 14:45:59.816810 IP 213.83.6.106.3270 > 85.159.14.110.443: F 1:1(0) ack 1 win 33304 14:45:59.816900 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 2 win 8326 14:45:59.818445 IP 85.159.14.110.443 > 213.83.6.106.3270: F 1:1(0) ack 2 win 8326 14:45:59.822859 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.415401 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 1 win 0 14:46:00.420082 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.420139 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.424772 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.424847 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.429065 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.429089 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.433247 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.433305 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.437641 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.437700 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.442408 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.442445 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.447231 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.447291 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.451525 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.451587 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.455957 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.456024 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.460666 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.460732 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:00.465092 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:00.465150 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 [...] 14:46:31.182624 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:31.182978 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.183006 IP 85.159.14.110.443 > 213.83.6.106.3270: . ack 1 win 65535 14:46:31.183146 IP 85.159.14.110.443 > 213.83.6.106.3270: F 1:1(0) ack 1 win 65535 14:46:31.183173 IP 85.159.14.110.443 > 213.83.6.106.3270: F 1:1(0) ack 1 win 65535 14:46:31.184038 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.184124 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.184157 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.184740 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.185174 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.186762 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.187366 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.187380 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 14:46:31.187573 IP 213.83.6.106.3270 > 85.159.14.110.443: . ack 2 win 33303 443 is a self written server, but it also happens with port 80 and sslproxy. The client is a telnet, which disconnects directly after connecting, so the disconnect is initiated from the client, which seems to be important for this problem to trigger. You can see that the FIN handshake completes and netstat on the client box shows the connection in TIME_WAIT. The server however has the connection still in ESTABLISHED state. What happens in the application code looks quite silly. I do a typical accept loop and then I process the data in a new thread. After my thread terminates and closes it's filedescriptor the select loop accepts the old connection again. This doesn't happen in every case but almost always. Finally after 30 seconds without data to read my newly created thread closes the zombie connection again. The question is why accept returns me a filedescriptor for a connection which was already returned and should have been closed? -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.