From owner-freebsd-questions@freebsd.org Fri Jul 26 12:31:25 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D9547BEE4F for ; Fri, 26 Jul 2019 12:31:25 +0000 (UTC) (envelope-from igorr@pochta-mx.canmos.ru) Received: from pochta-mx.canmos.ru (pochta-mx.canmos.ru [89.107.127.240]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 55DD38E1CD for ; Fri, 26 Jul 2019 12:31:25 +0000 (UTC) (envelope-from igorr@pochta-mx.canmos.ru) Received: from pochta-mx.canmos.ru (pochta-mx.canmos.ru [89.107.127.240]) by pochta-mx.canmos.ru (Postfix) with ESMTP id E3CEA2DBC3BD; Fri, 26 Jul 2019 15:28:44 +0300 (MSK) X-Spam-Checker-Version: SpamAssassin 3.3.2-pochta_20190408 (2011-06-06) on pochta.canmos.ru X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham version=3.3.2-pochta_20190408 X-Spam-Report: * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] Received: from pochta-mx.canmos.ru (pochta-mx.canmos.ru [89.107.127.240]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by pochta-mx.canmos.ru (Postfix) with ESMTPS; Fri, 26 Jul 2019 15:28:44 +0300 (MSK) Date: Fri, 26 Jul 2019 15:28:44 +0300 (MSK) From: "Igor V. Ruzanov" X-X-Sender: igorr@pochta.canmos.ru To: Paul Macdonald cc: FreeBSD Questions Subject: Re: Help:: Listen queue overflow killing servers In-Reply-To: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> Message-ID: References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-GPG-PUBLIC-KEY: 1024D/F433BDD5 2009-06-17 Igor V. Ruzanov X-GPG-FINGERPRINT: 5030 C793 4238 FAFF 827F 0E99 FDCE 63DD F433 BDD5 MIME-Version: 1.0 X-Rspamd-Queue-Id: 55DD38E1CD X-Spamd-Bar: ++++++++ Authentication-Results: mx1.freebsd.org; dmarc=fail reason="" header.from=canmos.ru (policy=none) X-Spamd-Result: default: False [8.62 / 15.00]; ARC_NA(0.00)[]; RSPAMD_URIBL(4.50)[ifdnrg.com]; URIBL_BLOCKED(0.00)[ifdnrg.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.95)[0.948,0]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; IP_SCORE(0.08)[ipnet: 89.107.120.0/21(0.23), asn: 39821(0.18), country: RU(0.01)]; NEURAL_SPAM_MEDIUM(1.00)[0.999,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: pochta-mx.canmos.ru]; CTYPE_MIXED_BOGUS(1.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(1.00)[1.000,0]; R_SPF_NA(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:39821, ipnet:89.107.120.0/21, country:RU]; RCVD_COUNT_TWO(0.00)[2]; GREYLIST(0.00)[pass,body]; DMARC_POLICY_SOFTFAIL(0.10)[canmos.ru : No valid SPF, No valid DKIM,none] X-Spam: Yes Content-Type: TEXT/PLAIN; charset=utf-8 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 12:31:25 -0000 You can increase TCP queue at system level: sysctl kern.ipc.somaxconn=65535 (its maximum value). But often its not enough because of application you use. So dig into application settings to increase the queue of TCP sockets to be handled in your system. On Fri, 26 Jul 2019, Paul Macdonald via freebsd-questions wrote: | |Hi, | |Over the past few months i've seen several boxes (4 or 5) become unresponsive |as a result of a Listen queue overflow state. | |Processes stack up, none are killable, all these are within jails and neither |the jail can be stopped nor the server rebooted (without a power cycle). | |All are on ZFS and are std apache/php/mysql servers with nothing too exotic. | |All on 12.0-RELEASE, i've only started seeing these issues recently, but it |feels like more and more. | |/var/log/messages shows tyically; | |    kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: 193 |already in queue awaiting acceptance (83 occurrences) | |netstat -Lan  shows | |tcp4 193/0/128                          x.x.x.x.443 |tcp4  193/0/128                          x.x.x.x.80 | |connections cannot be killed with tcpdrop ( except ssh which can!) | |All processes seem to be in Disk State ( many many apache processes but others |getting stuck too) | |www      60089    0.0 0.1  196588   78328  -  DJ   21:07         1:19.54 |/usr/local/sbin/httpd -DNOHTTPACCEPT |.. | |www      93713    0.0 0.0  183576   33164  -  DJ   23:57         0:00.01 |/usr/local/sbin/httpd -DNOHTTPACCEPT | |but no zombies.. | |last pid: 24773;  load averages:  0.00,  0.00, 0.00                       up |52+11:41:09  11:48:02 |918 processes: 1 running, 917 sleeping |CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle |Mem: 107M Active, 3729M Inact, 93G Wired, 27G Free |ARC: 79G Total, 54G MFU, 23G MRU, 243M Anon, 710M Header, 1615M Other |     73G Compressed, 191G Uncompressed, 2.60:1 Ratio |Swap: 4096M Total, 4096M Free | | |I'd appreciate any advice as at present it looks like my only option is to hard |power cycle these | |many thanks | |Paul. | | | |-- |------------------------- |Paul Macdonald |IFDNRG Ltd |Web and video hosting |------------------------- |t: 0131 5548070 |m: 07970339546 |e: paul@ifdnrg.com |w: http://www.ifdnrg.com |------------------------- |IFDNRG |40 Maritime Street |Edinburgh |EH6 6SA |---------------------------------------------------- | |Virtual Servers from £50.00pm |High specification Dedicated Servers from £150.00pm | |---------------------------------------------------- | |_______________________________________________ |freebsd-questions@freebsd.org mailing list |https://lists.freebsd.org/mailman/listinfo/freebsd-questions |To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" |