From owner-freebsd-questions@freebsd.org Fri Jul 26 12:47:27 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9983BBF2B5 for ; Fri, 26 Jul 2019 12:47:27 +0000 (UTC) (envelope-from chris@cretaforce.gr) Received: from smtp1.cretaforce.gr (smtp1.cretaforce.gr [159.69.10.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.cretaforce.gr", Issuer "RapidSSL RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D320F8EAD6 for ; Fri, 26 Jul 2019 12:47:26 +0000 (UTC) (envelope-from chris@cretaforce.gr) Received: from server1.cretaforce.gr (server1.cretaforce.gr [138.201.248.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.cretaforce.gr", Issuer "RapidSSL RSA CA 2018" (verified OK)) by smtp1.cretaforce.gr (Postfix) with ESMTPS id 0EE5B1FAD5 for ; Fri, 26 Jul 2019 15:39:57 +0300 (EEST) Received: from [192.168.2.2] (athedsl-279463.home.otenet.gr [85.73.139.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: chris@cretaforce.gr) by server1.cretaforce.gr (Postfix) with ESMTPSA id C1CD92730B for ; Fri, 26 Jul 2019 15:39:56 +0300 (EEST) From: Christos Chatzaras Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: Help:: Listen queue overflow killing servers Date: Fri, 26 Jul 2019 15:39:55 +0300 References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> To: FreeBSD Questions In-Reply-To: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> Message-Id: <8E871E29-53D5-4F88-A0B3-F1A3F966CF0B@cretaforce.gr> X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: D320F8EAD6 X-Spamd-Bar: ++++++++ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of chris@cretaforce.gr designates 159.69.10.232 as permitted sender) smtp.mailfrom=chris@cretaforce.gr X-Spamd-Result: default: False [8.01 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:159.69.10.232]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mail.cretaforce.gr]; RCVD_IN_DNSWL_LOW(-0.10)[232.10.69.159.list.dnswl.org : 127.0.5.1]; IP_SCORE(0.22)[ipnet: 159.69.0.0/16(2.93), asn: 24940(-1.83), country: DE(-0.01)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RSPAMD_URIBL(4.50)[ifdnrg.com]; ARC_NA(0.00)[]; URIBL_BLOCKED(0.00)[ifdnrg.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.90)[0.905,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; DMARC_NA(0.00)[cretaforce.gr]; NEURAL_SPAM_MEDIUM(1.00)[1.000,0]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; NEURAL_SPAM_LONG(1.00)[1.000,0]; GREYLIST(0.00)[pass,body]; RCVD_TLS_ALL(0.00)[] X-Spam: Yes X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 12:47:27 -0000 Do you have "sendfile" enabled in Apache? If yes disable it and see if = the issue is resolved. > On 26 Jul 2019, at 14:58, Paul Macdonald via freebsd-questions = wrote: >=20 >=20 > Hi, >=20 > Over the past few months i've seen several boxes (4 or 5) become = unresponsive as a result of a Listen queue overflow state. >=20 > Processes stack up, none are killable, all these are within jails and = neither the jail can be stopped nor the server rebooted (without a power = cycle). >=20 > All are on ZFS and are std apache/php/mysql servers with nothing too = exotic. >=20 > All on 12.0-RELEASE, i've only started seeing these issues recently, = but it feels like more and more. >=20 > /var/log/messages shows tyically; >=20 > kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: = 193 already in queue awaiting acceptance (83 occurrences) >=20 > netstat -Lan shows >=20 > tcp4 193/0/128 x.x.x.x.443 > tcp4 193/0/128 x.x.x.x.80 >=20 > connections cannot be killed with tcpdrop ( except ssh which can!) >=20 > All processes seem to be in Disk State ( many many apache processes = but others getting stuck too) >=20 > www 60089 0.0 0.1 196588 78328 - DJ 21:07 = 1:19.54 /usr/local/sbin/httpd -DNOHTTPACCEPT > .. >=20 > www 93713 0.0 0.0 183576 33164 - DJ 23:57 = 0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT >=20 > but no zombies.. >=20 > last pid: 24773; load averages: 0.00, 0.00, 0.00 = up 52+11:41:09 11:48:02 > 918 processes: 1 running, 917 sleeping > CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle > Mem: 107M Active, 3729M Inact, 93G Wired, 27G Free > ARC: 79G Total, 54G MFU, 23G MRU, 243M Anon, 710M Header, 1615M Other > 73G Compressed, 191G Uncompressed, 2.60:1 Ratio > Swap: 4096M Total, 4096M Free >=20 >=20 > I'd appreciate any advice as at present it looks like my only option = is to hard power cycle these >=20 > many thanks >=20 > Paul. >=20 >=20 >=20 > --=20 > ------------------------- > Paul Macdonald > IFDNRG Ltd > Web and video hosting > ------------------------- > t: 0131 5548070 > m: 07970339546 > e: paul@ifdnrg.com > w: http://www.ifdnrg.com > ------------------------- > IFDNRG > 40 Maritime Street > Edinburgh > EH6 6SA > ---------------------------------------------------- >=20 > Virtual Servers from =C2=A350.00pm > High specification Dedicated Servers from =C2=A3150.00pm >=20 > ---------------------------------------------------- >=20 > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to = "freebsd-questions-unsubscribe@freebsd.org"