From owner-freebsd-stable@FreeBSD.ORG Wed May 28 22:33:18 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2E6B1065674 for ; Wed, 28 May 2008 22:33:18 +0000 (UTC) (envelope-from rblayzor.bulk@inoc.net) Received: from mx1-a.inoc.net (mx1-a.inoc.net [64.246.131.30]) by mx1.freebsd.org (Postfix) with ESMTP id 8E10D8FC1D for ; Wed, 28 May 2008 22:33:18 +0000 (UTC) (envelope-from rblayzor.bulk@inoc.net) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=inoc.net; h=Received:From:To:Subject:Date; b=ZuWmLH5fTHYxBjQoZk/FlYbc9fsz3RgEwdCYXjNMG0EeT/nYff3OrkWdW/ANuO8kOQJVuZ5zCRIkr4+ekIJlHcy1unTltWmBGo/8KwZ3PtQJkJHgs2jX9jmnuTGcFTHBE+V/ttpzFKoyc0oGqoFp+yzlyu4Pziw8IetrdrDesM0=; Received: from [172.16.0.199] (cpe-67-240-119-200.nycap.res.rr.com [67.240.119.200]) by mx1-a.inoc.net (build v8.3.29) with ESMTP id 148351714-1941382 for ; Wed, 28 May 2008 22:13:07 +0000 (UTC) Message-Id: From: Robert Blayzor To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v924) Date: Wed, 28 May 2008 18:13:04 -0400 X-Mailer: Apple Mail (2.924) Subject: Sockets stuck in FIN_WAIT_1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 May 2008 22:33:18 -0000 I have a rather busy Apache 2.2 server; tons of small & some large requests. It's a standard Dell 2650 server using the bge (broadcom) network driver. I seem to have a rather strange problem where after just a day or so Apache just stops processing new connections. You can connect to port 80, but trying to get Apache to process any data just hangs. There is nothing strange in dmesg or in /var/log/messages. The server has plenty free available physical RAM, swap is untouched, CPU load is low, etc. Apache is setup to handle a max of 100 clients using prefork model. If I stop and restart Apache, it does not help. What I do notice is 1000's of sockets stuck in "FIN_WAIT_1" in netstat: [web0:~] netstat -an | grep FIN_WAIT | wc -l 1827 These stick around forever. Some eventually trickle away after hours, but the only thing that appears to fix it is to reboot the server. Then all is fine for another day or so. I've tried just about every tuning trick out there but to no eval. I can mitigate the problem by increasing available socket buffs and decreasing the tcp.sendspace. I've tried different versions of Apache and I've tried with and without the accf_http kernel filter. Here is what I have on the server now: sysctl.conf: kern.maxfiles=65535 kern.maxfilesperproc=16384 kern.ipc.maxsockbuf=4194304 kern.ipc.somaxconn=1024 net.inet.tcp.sendspace=8192 net.inet.tcp.recvspace=8192 net.inet.tcp.keepidle=900000 net.inet.tcp.keepintvl=30000 net.inet.tcp.msl=5000 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.tcp.inflight_enable=1 and loader.conf accf_http_load="YES" kern.ipc.nmbclusters=32768 net.inet.tcp.tcbhashsize=4096 kern.ipc.maxsockets=131072 ipfw: 00200 allow tcp from any to me 80 setup 00200 allow icmp from any to me icmptype 0,3,8,11 00200 deny log ip from any to me ifconfig: bge0: flags=8843 mtu 1500 options=3 inet 1.2.3.4 netmask 0xfffffff8 broadcast 5.6.7.8 ether 00:06:5b:f7:c8:7b media: Ethernet autoselect (1000baseTX ) Any ideas would be greatly appreciated. -- Robert Blayzor, BOFH INOC, LLC rblayzor@inoc.net http://www.inoc.net/~rblayzor/