Date: Fri, 26 Jul 2019 12:58:45 +0100 From: Paul Macdonald <paul@ifdnrg.com> To: FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Help:: Listen queue overflow killing servers Message-ID: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com>
next in thread | raw e-mail | index | archive | help
Hi, Over the past few months i've seen several boxes (4 or 5) become unresponsive as a result of a Listen queue overflow state. Processes stack up, none are killable, all these are within jails and neither the jail can be stopped nor the server rebooted (without a power cycle). All are on ZFS and are std apache/php/mysql servers with nothing too exotic. All on 12.0-RELEASE, i've only started seeing these issues recently, but it feels like more and more. /var/log/messages shows tyically; kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: 193 already in queue awaiting acceptance (83 occurrences) netstat -Lan shows tcp4 193/0/128 x.x.x.x.443 tcp4 193/0/128 x.x.x.x.80 connections cannot be killed with tcpdrop ( except ssh which can!) All processes seem to be in Disk State ( many many apache processes but others getting stuck too) www 60089 0.0 0.1 196588 78328 - DJ 21:07 1:19.54 /usr/local/sbin/httpd -DNOHTTPACCEPT ..<snoip> www 93713 0.0 0.0 183576 33164 - DJ 23:57 0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT but no zombies.. last pid: 24773; load averages: 0.00, 0.00, 0.00 up 52+11:41:09 11:48:02 918 processes: 1 running, 917 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 107M Active, 3729M Inact, 93G Wired, 27G Free ARC: 79G Total, 54G MFU, 23G MRU, 243M Anon, 710M Header, 1615M Other 73G Compressed, 191G Uncompressed, 2.60:1 Ratio Swap: 4096M Total, 4096M Free I'd appreciate any advice as at present it looks like my only option is to hard power cycle these many thanks Paul. -- ------------------------- Paul Macdonald IFDNRG Ltd Web and video hosting ------------------------- t: 0131 5548070 m: 07970339546 e: paul@ifdnrg.com w: http://www.ifdnrg.com ------------------------- IFDNRG 40 Maritime Street Edinburgh EH6 6SA ---------------------------------------------------- Virtual Servers from £50.00pm High specification Dedicated Servers from £150.00pm ----------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3a62375a-432c-3533-a7bc-e5573c26fa9c>