From owner-freebsd-stable@FreeBSD.ORG Wed Mar 4 01:41:23 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4ADCA262 for ; Wed, 4 Mar 2015 01:41:23 +0000 (UTC) Received: from www81.your-server.de (www81.your-server.de [213.133.104.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B7DCBF5 for ; Wed, 4 Mar 2015 01:41:22 +0000 (UTC) Received: from [24.134.170.44] (helo=michael-think.fritz.box) by www81.your-server.de with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1) (envelope-from ) id 1YSyJE-0005a1-Jg; Wed, 04 Mar 2015 02:41:12 +0100 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: "freebsd-stable@freebsd.org" , "Rumen Telbizov" Subject: Re: Stale TIME_WAIT tcp connections References: Date: Wed, 04 Mar 2015 02:41:07 +0100 MIME-Version: 1.0 Content-Transfer-Encoding: Quoted-Printable From: "Michael Ross" Message-ID: In-Reply-To: User-Agent: Opera Mail/1.0 (Win32) X-Authenticated-Sender: gmx@ross.cx X-Virus-Scanned: Clear (ClamAV 0.98.5/20142/Wed Mar 4 00:54:52 2015) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 01:41:23 -0000 On Wed, 04 Mar 2015 01:36:18 +0100, Rumen Telbizov = = wrote: > Hello everyone, > > We have a server running 9.3-RELEASE which is exhibiting a high number= of > TIME_WAIT tcp connections which are NOT being recycled. That is, netst= at > reports them over and over again, no matter how long we wait for them = to = > be > flushed out. Currently this server has been out of rotation for a coup= le = > of > hours and I still see the same tcp sockets there. Overall we have: > > # netstat -na | grep TIME_WAIT | wc -l > *30066* > > Tracking one particular TCP socket in TIME_WAIT proves that it stays = > there > all the time. > > Another observation is that pfctl shows a very large number of state > entries, even after pfctl -F all, or disable/enable sequence. > > # pfctl -si > State Table Total Rate > current entries *59280* > > At the same time though: > > # pfctl -ss | wc -l > 18 > > After the problem was discovered we tried tweaking the following setti= ngs > without any luck: > > net.inet.tcp.fast_finwait2_recycle=3D1 > net.inet.tcp.finwait2_timeout=3D5000 > net.inet.tcp.maxtcptw=3D50000 > net.inet.tcp.msl=3D100 > > =E2=80=8BSo it seems like this system is "stuck" and =E2=80=8Bdoesn't = recycle those TCP > sockets. Again, the machine is out of rotation and not actively accept= ing > any traffic. I will keep it like that in case further investigation is= > required. Please do let me know if there's anything else you'd like to= = > know > from the state of the machine or something I could try. > > =E2=80=8BRegards, Are you using any IPSEC? I observed something similar a while back, haven't checked again since i= = reported this. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D194690 Affected 9.2, too. Michael