Date: Wed, 28 May 2008 18:13:04 -0400 From: Robert Blayzor <rblayzor.bulk@inoc.net> To: freebsd-stable@freebsd.org Subject: Sockets stuck in FIN_WAIT_1 Message-ID: <B42F9BDF-1E00-45FF-BD88-5A07B5B553DC@inoc.net>
next in thread | raw e-mail | index | archive | help
I have a rather busy Apache 2.2 server; tons of small & some large
requests. It's a standard Dell 2650 server using the bge (broadcom)
network driver.
I seem to have a rather strange problem where after just a day or so
Apache just stops processing new connections. You can connect to port
80, but trying to get Apache to process any data just hangs. There is
nothing strange in dmesg or in /var/log/messages.
The server has plenty free available physical RAM, swap is untouched,
CPU load is low, etc. Apache is setup to handle a max of 100 clients
using prefork model.
If I stop and restart Apache, it does not help.
What I do notice is 1000's of sockets stuck in "FIN_WAIT_1" in netstat:
[web0:~] netstat -an | grep FIN_WAIT | wc -l
1827
These stick around forever. Some eventually trickle away after hours,
but the only thing that appears to fix it is to reboot the server.
Then all is fine for another day or so. I've tried just about every
tuning trick out there but to no eval. I can mitigate the problem by
increasing available socket buffs and decreasing the tcp.sendspace.
I've tried different versions of Apache and I've tried with and
without the accf_http kernel filter.
Here is what I have on the server now:
sysctl.conf:
kern.maxfiles=65535
kern.maxfilesperproc=16384
kern.ipc.maxsockbuf=4194304
kern.ipc.somaxconn=1024
net.inet.tcp.sendspace=8192
net.inet.tcp.recvspace=8192
net.inet.tcp.keepidle=900000
net.inet.tcp.keepintvl=30000
net.inet.tcp.msl=5000
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.inflight_enable=1
and loader.conf
accf_http_load="YES"
kern.ipc.nmbclusters=32768
net.inet.tcp.tcbhashsize=4096
kern.ipc.maxsockets=131072
ipfw:
00200 allow tcp from any to me 80 setup
00200 allow icmp from any to me icmptype 0,3,8,11
00200 deny log ip from any to me
ifconfig:
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
inet 1.2.3.4 netmask 0xfffffff8 broadcast 5.6.7.8
ether 00:06:5b:f7:c8:7b
media: Ethernet autoselect (1000baseTX <full-duplex>)
Any ideas would be greatly appreciated.
--
Robert Blayzor, BOFH
INOC, LLC
rblayzor@inoc.net
http://www.inoc.net/~rblayzor/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B42F9BDF-1E00-45FF-BD88-5A07B5B553DC>