Date: Fri, 26 Jul 2019 14:52:08 +0200 From: Robert Schulze <rs@bytecamp.net> To: FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: Help:: Listen queue overflow killing servers Message-ID: <92866b76-5f11-2523-cc8f-0d92cc91a50e@bytecamp.net> In-Reply-To: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions: > Over the past few months i've seen several boxes (4 or 5) become > unresponsive as a result of a Listen queue overflow state. > > Processes stack up, none are killable, all these are within jails and > neither the jail can be stopped nor the server rebooted (without a power > cycle). > > All are on ZFS and are std apache/php/mysql servers with nothing too > exotic. > > All on 12.0-RELEASE, i've only started seeing these issues recently, but > it feels like more and more. > > /var/log/messages shows tyically; > > kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: > 193 already in queue awaiting acceptance (83 occurrences) > > netstat -Lan shows > > tcp4 193/0/128 x.x.x.x.443 > tcp4 193/0/128 x.x.x.x.80 > > connections cannot be killed with tcpdrop ( except ssh which can!) > > All processes seem to be in Disk State ( many many apache processes but > others getting stuck too) > > www 60089 0.0 0.1 196588 78328 - DJ 21:07 1:19.54 > /usr/local/sbin/httpd -DNOHTTPACCEPT > ..<snoip> > > www 93713 0.0 0.0 183576 33164 - DJ 23:57 0:00.01 > /usr/local/sbin/httpd -DNOHTTPACCEPT > I think, these processes waiting for disk i/o are actually your problem. Since they cannot answer further requests, they run into the listen queue overflow. You should check the processes with procstat: list kernel threads: # procstat -kk <PID> list open files: # procstat -f <PID> with kind regards, Robert Schulze
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?92866b76-5f11-2523-cc8f-0d92cc91a50e>