From owner-freebsd-stable@FreeBSD.ORG Wed Mar 4 00:36:19 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 001D5BF6 for ; Wed, 4 Mar 2015 00:36:18 +0000 (UTC) Received: from mail-ig0-x230.google.com (mail-ig0-x230.google.com [IPv6:2607:f8b0:4001:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA74F38C for ; Wed, 4 Mar 2015 00:36:18 +0000 (UTC) Received: by igbhl2 with SMTP id hl2so32818033igb.3 for ; Tue, 03 Mar 2015 16:36:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=zzkmJRbKy2YBhnOVrxjeDt06MmnigBn99DRxrcPAEjc=; b=wDrr8ASmTnpySyqXsUu7i1YnjtkdvYvMMsa/qMNrYiLHuwmSaigYMGdKYBjkcsxaMf IFzdOMsb2yyqR6kRvPKnm5R7PfhOiavDdmWztaytFr50u7co45TZElTN1uTE+BlkQCiw TTqYmmYoygUb0GHBR4M87JJrcSpHLYh20+08NCXAmb7y6fH7eMr5d1nCvQBVPowRn6/W D/fo/ed3EWLOBWV7SUfwnDhXt2eqgP/KNK6sIxUKhE5507t3re942lhvWbsjpMmbXIfZ tWRKZ+TYjE9lWbydrRGgCy0g6V59Dg9BA9kbyAV67j6DDx2oFRpgQ5XqJz46x3RYz/tA pi8g== MIME-Version: 1.0 X-Received: by 10.107.25.72 with SMTP id 69mr6687441ioz.44.1425429378124; Tue, 03 Mar 2015 16:36:18 -0800 (PST) Received: by 10.50.243.38 with HTTP; Tue, 3 Mar 2015 16:36:18 -0800 (PST) Date: Tue, 3 Mar 2015 16:36:18 -0800 Message-ID: Subject: Stale TIME_WAIT tcp connections From: Rumen Telbizov To: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Mar 2015 00:36:19 -0000 Hello everyone, We have a server running 9.3-RELEASE which is exhibiting a high number of TIME_WAIT tcp connections which are NOT being recycled. That is, netstat reports them over and over again, no matter how long we wait for them to be flushed out. Currently this server has been out of rotation for a couple of hours and I still see the same tcp sockets there. Overall we have: # netstat -na | grep TIME_WAIT | wc -l *30066* Tracking one particular TCP socket in TIME_WAIT proves that it stays there all the time. Another observation is that pfctl shows a very large number of state entries, even after pfctl -F all, or disable/enable sequence. # pfctl -si State Table Total Rate current entries *59280* At the same time though: # pfctl -ss | wc -l 18 After the problem was discovered we tried tweaking the following settings without any luck: net.inet.tcp.fast_finwait2_recycle=3D1 net.inet.tcp.finwait2_timeout=3D5000 net.inet.tcp.maxtcptw=3D50000 net.inet.tcp.msl=3D100 =E2=80=8BSo it seems like this system is "stuck" and =E2=80=8Bdoesn't recyc= le those TCP sockets. Again, the machine is out of rotation and not actively accepting any traffic. I will keep it like that in case further investigation is required. Please do let me know if there's anything else you'd like to know from the state of the machine or something I could try. =E2=80=8BRegards, --=20 Rumen Telbizov Unix Systems Administrator