From owner-freebsd-questions@freebsd.org Fri Jul 26 12:15:32 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8C083BE660 for ; Fri, 26 Jul 2019 12:15:32 +0000 (UTC) (envelope-from paul@ifdnrg.com) Received: from outbound.ifdnrg.com (outbound.ifdnrg.com [193.200.98.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "outbound.ifdnrg.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 384298D7B5 for ; Fri, 26 Jul 2019 12:15:28 +0000 (UTC) (envelope-from paul@ifdnrg.com) Received: from [192.168.0.210] (38.156-30-62.static.virginmediabusiness.co.uk [62.30.156.38]) (authenticated bits=0) by outbound.ifdnrg.com (8.15.2/8.15.2) with ESMTPSA id x6QBwioA009627 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 26 Jul 2019 12:58:44 +0100 (BST) (envelope-from paul@ifdnrg.com) X-Authentication-Warning: outbound.ifdnrg.com: Host 38.156-30-62.static.virginmediabusiness.co.uk [62.30.156.38] claimed to be [192.168.0.210] To: FreeBSD Questions From: Paul Macdonald Subject: Help:: Listen queue overflow killing servers Message-ID: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> Date: Fri, 26 Jul 2019 12:58:45 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 Content-Language: en-GB X-Rspamd-Queue-Id: 384298D7B5 X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.36 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:193.200.98.0/23]; HAS_XAW(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[ifdnrg.com:+]; DMARC_POLICY_ALLOW(0.00)[ifdnrg.com,quarantine]; MX_GOOD(-0.01)[as1.ifdnrg.com,as3.ifdnrg.com]; RCVD_IN_DNSWL_LOW(-0.10)[22.98.200.193.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:20860, ipnet:193.200.98.0/23, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RSPAMD_URIBL(4.50)[ifdnrg.com]; R_DKIM_ALLOW(0.00)[ifdnrg.com:s=ifdnrg-default]; URIBL_BLOCKED(0.00)[ifdnrg.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.48)[0.481,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; NEURAL_HAM_MEDIUM(-0.42)[-0.419,0]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[ifdnrg.com.dwl.dnswl.org : 127.0.5.1]; BAD_REP_POLICIES(0.10)[]; NEURAL_SPAM_LONG(0.04)[0.042,0]; IP_SCORE(-1.13)[ipnet: 193.200.98.0/23(-4.83), asn: 20860(-0.75), country: GB(-0.08)]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 12:15:32 -0000 Hi, Over the past few months i've seen several boxes (4 or 5) become unresponsive as a result of a Listen queue overflow state. Processes stack up, none are killable, all these are within jails and neither the jail can be stopped nor the server rebooted (without a power cycle). All are on ZFS and are std apache/php/mysql servers with nothing too exotic. All on 12.0-RELEASE, i've only started seeing these issues recently, but it feels like more and more. /var/log/messages shows tyically;     kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: 193 already in queue awaiting acceptance (83 occurrences) netstat -Lan  shows tcp4 193/0/128                          x.x.x.x.443 tcp4  193/0/128                          x.x.x.x.80 connections cannot be killed with tcpdrop ( except ssh which can!) All processes seem to be in Disk State ( many many apache processes but others getting stuck too) www      60089    0.0 0.1  196588   78328  -  DJ   21:07         1:19.54 /usr/local/sbin/httpd -DNOHTTPACCEPT .. www      93713    0.0 0.0  183576   33164  -  DJ   23:57         0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT but no zombies.. last pid: 24773;  load averages:  0.00,  0.00, 0.00                       up 52+11:41:09  11:48:02 918 processes: 1 running, 917 sleeping CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle Mem: 107M Active, 3729M Inact, 93G Wired, 27G Free ARC: 79G Total, 54G MFU, 23G MRU, 243M Anon, 710M Header, 1615M Other      73G Compressed, 191G Uncompressed, 2.60:1 Ratio Swap: 4096M Total, 4096M Free I'd appreciate any advice as at present it looks like my only option is to hard power cycle these many thanks Paul. -- ------------------------- Paul Macdonald IFDNRG Ltd Web and video hosting ------------------------- t: 0131 5548070 m: 07970339546 e: paul@ifdnrg.com w: http://www.ifdnrg.com ------------------------- IFDNRG 40 Maritime Street Edinburgh EH6 6SA ---------------------------------------------------- Virtual Servers from £50.00pm High specification Dedicated Servers from £150.00pm ----------------------------------------------------