From owner-freebsd-bugs@FreeBSD.ORG Wed Dec 16 14:50:01 2009 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 809AF10656A5 for ; Wed, 16 Dec 2009 14:50:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 439098FC1D for ; Wed, 16 Dec 2009 14:50:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nBGEo1pR094265 for ; Wed, 16 Dec 2009 14:50:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nBGEo1KW094264; Wed, 16 Dec 2009 14:50:01 GMT (envelope-from gnats) Resent-Date: Wed, 16 Dec 2009 14:50:01 GMT Resent-Message-Id: <200912161450.nBGEo1KW094264@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Steve Polyack Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA4DC106566B for ; Wed, 16 Dec 2009 14:42:29 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id B8F3A8FC1C for ; Wed, 16 Dec 2009 14:42:29 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id nBGEgTSl053908 for ; Wed, 16 Dec 2009 14:42:29 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id nBGEgTOI053907; Wed, 16 Dec 2009 14:42:29 GMT (envelope-from nobody) Message-Id: <200912161442.nBGEgTOI053907@www.freebsd.org> Date: Wed, 16 Dec 2009 14:42:29 GMT From: Steve Polyack To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: bin/141683: nfsd(8) appears to hold onto closed connections indefinitely X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Dec 2009 14:50:01 -0000 >Number: 141683 >Category: bin >Synopsis: nfsd(8) appears to hold onto closed connections indefinitely >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Dec 16 14:50:00 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Steve Polyack >Release: FreeBSD 7.1-RELEASE amd64 >Organization: Collaborative Fusion, Inc. >Environment: FreeBSD ***** 7.1-RELEASE-p5 FreeBSD 7.1-RELEASE-p5 #3: Fri May 22 07:56:55 EDT 2009 root@*****:/usr/obj/export/src-RELENG_7_1/src/sys/POWEREDGE amd64 >Description: A server running FreeBSD 7.1-RELEASE-p5 which acts as an nfs server does not seem to ever let go of closed connections. The connections in question are from a service monitoring daemon (Nagios check_tcp). Telnet connections which are then broken will also leave connections lying around in the CLOSE_WAIT state. $ netstat -tan | grep CLOSE_WAIT |grep 2049 |wc -l 454 We've witnessed the server build up to over 7000 connections kept in the CLOSE_WAIT state after a few weeks. Restarting nfsd(8) will cause all of these held connections to disappear. >How-To-Repeat: Start nfsd(8) with the -t switch to serve TCP NFS. Open several telnet connections to the NFS server on port 2049 and close them immediately after connection (or use Nagios to perform simple TCP port monitoring): nfs-server$ netstat -tan | grep CLOSE_WAIT |grep 2049 |wc -l 455 $ telnet nfs-server 2049 Connected to nfs-server Escape character is '^]'. ^] telnet> q Connection closed. $ telnet nfs-server 2049 Connected to nfs-server Escape character is '^]'. ^] telnet> q Connection closed. $ telnet nfs-server 2049 Connected to nfs-server Escape character is '^]'. ^] telnet> q Connection closed. $ netstat -tan | grep CLOSE_WAIT |grep 2049 |wc -l 458 Wait minutes, hours, days, or weeks and see that the connections are still listed in CLOSE_WAIT by netstat. Eventually restart nfsd(8) and witness the number of connections drop significantly: $ sudo /etc/rc.d/nfsd restart Stopping nfsd. Starting nfsd. $ netstat -tan | grep CLOSE_WAIT |grep 2049 |wc -l 0 >Fix: >Release-Note: >Audit-Trail: >Unformatted: