From owner-freebsd-net@FreeBSD.ORG Wed Jun 25 18:46:46 2008 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C72CC1065672 for ; Wed, 25 Jun 2008 18:46:46 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.187]) by mx1.freebsd.org (Postfix) with ESMTP id 35BB78FC16 for ; Wed, 25 Jun 2008 18:46:45 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: by fk-out-0910.google.com with SMTP id k31so3401840fkk.11 for ; Wed, 25 Jun 2008 11:46:44 -0700 (PDT) Received: by 10.82.185.3 with SMTP id i3mr662100buf.27.1214417846331; Wed, 25 Jun 2008 11:17:26 -0700 (PDT) Received: by 10.82.185.13 with HTTP; Wed, 25 Jun 2008 11:17:26 -0700 (PDT) Message-ID: Date: Wed, 25 Jun 2008 21:17:26 +0300 From: "Vlad GALU" To: "Ali Niknam" In-Reply-To: <486283B0.3060805@transip.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <486283B0.3060805@transip.nl> Cc: net@freebsd.org Subject: Re: FreeBSD 7.0: sockets stuck in CLOSED state... X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jun 2008 18:46:46 -0000 On 6/25/08, Ali Niknam wrote: > Dear All, > > Recently i've been upgrading some of my machines from FreeBSD 6.x amd64 to > FreeBSD 7.0 amd64. > > After upgrading I noticed a weird error/bug. It seems that after several > thousand TCP connections some seem to hang in 'CLOSED' state. > > netstat -n gives: > ... > tcp4 0 0 1.2.3.4.* 4.5.6.7.42149 CLOSED > tcp4 39 0 1.2.3.4.* 4.5.6.7.54103 CLOSED > tcp4 35 0 1.2.3.4.* 4.5.6.7.41718 CLOSED > tcp4 38 0 1.2.3.4.* 4.5.6.7.55618 CLOSED > tcp4 41 0 1.2.3.4.* 4.5.6.7.44230 CLOSED > tcp4 39 0 1.2.3.4.* 4.5.6.7.49439 CLOSED > ... > > These never go away; they gradually increase and increase until the > application starts giving errors (probably because some socket or > filedescriptor limit is reached). When the application is killed these > entries disappear. > > The application in question is a self written DNS server, multithreaded, > and running fine for years without any troubles on both BSD 5.x as well as > 6.x. Also 32bits as well as 64bits on 6.x. > > Ofcourse that doesn't mean that the application is error free, however, > after doing extensive testing I really can not find anything wrong with the > application itself, so I'm thinking maybe there's a change somewhere that > causes this? I know that tcp/network has been completely redone... > > What basically happens in the application is this: > - one main tcp thread runs an infinite while loop waiting for new > connections to arrive > - as soon as one arrives a new thread is spawned that handles the newly > created stream > - it reads some bytes, writes some bytes, then closes it > - thread exits > > What appears to happen is this: after the new thread is spawned it tries to > read 2 bytes (DNS tcp length information). It gets back 0 bytes (EOF) and > therefore closes the sockets and calls pthread_exit. However in netstat that > same stream oftenly appears to have bytes 'stuck' in the in queue... > > I really can't see how this can cause hanging sockets in 'CLOSED' state. > Even if the incoming queue isnt read entirely a call to close should close > it. Also I really can't find any documentation in netstat, or elsewhere, > about the 'CLOSED' state... > > > Any help would greatly be appreciated! > > > Kind Regards, > > > Ali Niknam > _______________________________________________ This looks like an issue we used to have at work, where a streaming application suddenly started getting kevents for sockets that had been already closed. While that was happening, a netstat output looked just like yours. We never tracked it down, as we moved to other projects :( -- ~/.signature: no such file or directory