From owner-freebsd-questions@freebsd.org Fri Jul 26 16:57:25 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 81AC8A4BC1 for ; Fri, 26 Jul 2019 16:57:25 +0000 (UTC) (envelope-from paul@ifdnrg.com) Received: from outbound.ifdnrg.com (outbound.ifdnrg.com [193.200.98.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "outbound.ifdnrg.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 992DC715D2 for ; Fri, 26 Jul 2019 16:57:24 +0000 (UTC) (envelope-from paul@ifdnrg.com) Received: from [192.168.0.210] (38.156-30-62.static.virginmediabusiness.co.uk [62.30.156.38]) (authenticated bits=0) by outbound.ifdnrg.com (8.15.2/8.15.2) with ESMTPSA id x6QGvJfW019685 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 26 Jul 2019 17:57:19 +0100 (BST) (envelope-from paul@ifdnrg.com) X-Authentication-Warning: outbound.ifdnrg.com: Host 38.156-30-62.static.virginmediabusiness.co.uk [62.30.156.38] claimed to be [192.168.0.210] Subject: Re: Help:: Listen queue overflow killing servers To: freebsd-questions@freebsd.org References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> <2b10f991-bc95-ae31-18e2-95ae943ac527@holgerdanske.com> From: Paul Macdonald Message-ID: <2798d3f3-9689-111c-e061-1f6f66d78e03@ifdnrg.com> Date: Fri, 26 Jul 2019 17:57:21 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <2b10f991-bc95-ae31-18e2-95ae943ac527@holgerdanske.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-Rspamd-Queue-Id: 992DC715D2 X-Spamd-Bar: / X-Spamd-Result: default: False [0.26 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:193.200.98.0/23]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; MX_GOOD(-0.01)[cached: as1.ifdnrg.com]; DKIM_TRACE(0.00)[ifdnrg.com:+]; DMARC_POLICY_ALLOW(0.00)[ifdnrg.com,quarantine]; NEURAL_HAM_SHORT(-0.85)[-0.850,0]; RCVD_IN_DNSWL_LOW(-0.10)[22.98.200.193.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:20860, ipnet:193.200.98.0/23, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RSPAMD_URIBL(4.50)[ifdnrg.com]; R_DKIM_ALLOW(0.00)[ifdnrg.com:s=ifdnrg-default]; URIBL_BLOCKED(0.00)[ifdnrg.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.40)[-0.402,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; NEURAL_HAM_MEDIUM(-0.77)[-0.771,0]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[ifdnrg.com.dwl.dnswl.org : 127.0.5.1]; BAD_REP_POLICIES(0.10)[]; IP_SCORE(-1.11)[ipnet: 193.200.98.0/23(-4.70), asn: 20860(-0.75), country: GB(-0.08)]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 16:57:25 -0000 On 26/07/2019 17:11, David Christensen wrote: > On 7/26/19 4:58 AM, Paul Macdonald via freebsd-questions wrote: >> Over the past few months i've seen several boxes (4 or 5) become >> unresponsive as a result of a Listen queue overflow state. > >> All are on ZFS and are std apache/php/mysql servers with nothing too >> exotic. > >> /var/log/messages shows tyically; >> >>      kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue >> overflow: 193 already in queue awaiting acceptance (83 occurrences) >> >> netstat -Lan  shows >> >> tcp4 193/0/128                          x.x.x.x.443 >> tcp4  193/0/128                          x.x.x.x.80 > > > What Apache/ PHP/ MySQL applications?  Did you write them?  If not, > who did?  Is everything up to date?  Have you filed bug reports? > > > Do the applications have logging or debugging capabilities?  Have you > enabled them?  What do they say?  Where is the blockage? Deadlock? > > These were on servers with multiple vhosts, often running wordpress , but in one instance not ( which had custom software we wrote inhouse , but thats been in production for 19 years without this issue!) I suspect it's too low level for application level debugging, all i know so far is:                 - servers become unresponsive, Listen queue overflow messages in /var/log/messages                 - unable to quit jails or even shutdown,  tcpdrop doesn't work (everything in CLOSE_WAIT)                 - On the occasion today ( and i can;t be 100% sure, but i siuspect always) , all the apache processes were in disk wait state,  but this was on a big new box, with a very tiny site, ( on NVMe)                All servers on FBSD12, with zfs and apache is within an (ezjail)                 Multiple load patterns, but 2 out of the 5ish issues don't make much sense as theere would have been very little load.                 Non reproducible, have sieged a couple of the affected boxes with no effect ( and logs on a couple of boxes show no intersting traffic, just normal)                         - siege -c 255 -r 2                         (pretty stressful)                     (target server does now something in netstat queues ,  0-100/512  but apache stays out of disk wait , siege is (un) sucessfull as target copes fine                 run multiple times , no problem, and have now generated about 100,000 lines more in apache log that i saw after the server went down today  ( (6600 hits to a 16C/32T  + 128GB + NVme machine went down with this earlier)                I've just hit it with 255 concurrent users over a period of 20 mins, and it doesn;t blink                 so doesn;t look like its load..... ( and that would have shown up in the logs anyway) > David > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" > -- ------------------------- Paul Macdonald IFDNRG Ltd Web and video hosting ------------------------- t: 0131 5548070 m: 07970339546 e: paul@ifdnrg.com w: http://www.ifdnrg.com ------------------------- IFDNRG 40 Maritime Street Edinburgh EH6 6SA ---------------------------------------------------- Virtual Servers from £50.00pm High specification Dedicated Servers from £150.00pm ----------------------------------------------------