From owner-freebsd-questions@freebsd.org Fri Jul 26 12:52:19 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6DD2BBF441 for ; Fri, 26 Jul 2019 12:52:19 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1FC2F8F01E for ; Fri, 26 Jul 2019 12:52:18 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id F3E1F55859; Fri, 26 Jul 2019 14:52:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=subject:to:references:from:message-id:date:mime-version:in-reply-to:content-type:content-transfer-encoding; s=20140709; bh=Xd/9K/YX6nVir1D8D+A9/ASiDrU=; b=i7TAPADgJHphCi8TLiG5HHmZMy0OJOWWxkcXjd1bbSKAP62YjErpEIP+lGrjCxC4gdPSCPG9nrsmKObKvyW6nbzDxLZpSwv2D06lIobXr4Hzg6us/7PVIJlPmD0DCysIyeywZsTlryGckk5ne7FeC7mAJiX4j35h6BJglH7CZI4= Received: from mail.bytecamp.net (mail.bytecamp.net [212.204.60.9]) by mxout01.bytecamp.net (Postfix) with ESMTP id BC0D055856 for ; Fri, 26 Jul 2019 14:52:08 +0200 (CEST) Received: (qmail 1955 invoked from network); 26 Jul 2019 14:52:08 +0200 Received: from unknown (HELO ?192.168.3.59?) (rs%bytecamp.net@80.84.212.123) by mail.bytecamp.net with ESMTPS (AES128-SHA encrypted); 26 Jul 2019 14:52:08 +0200 Subject: Re: Help:: Listen queue overflow killing servers To: FreeBSD Questions References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> From: Robert Schulze Openpgp: preference=signencrypt Autocrypt: addr=rs@bytecamp.net; prefer-encrypt=mutual; keydata= mQENBFRpvvcBCACiXR5ENQUFzwmQCyyQmpG3ppKXS5GamZHZSl6dr+yeTRHW5XJ+46ltis64 U/i1V0DhBD9kHjAvwZvWO+BXVj1kf1GFf286WoUnFvRK6oLRtaqdZDDu+AA3EcvvryRzlkE9 uSBXkg+4ObhxhaLhjWowDBhmAyi3QOAtPpw9tme4c0ktoKMMlkryTAwPXw7sJi7TGJXv088T z2C2eMBKNKwyCWmGsP+0UoZRxO0A7qCU1kIGvsf7N9sYFD3OI9NaYZHvN8g7x7NTInz77a2o /u1EFYcQ6UbphP8AhUX9phvQLsZr0w0eCFQ+kJqtHWdesRckQcpDo8eXDQOvZH5vxErzABEB AAG0IFJvYmVydCBTY2h1bHplIDxyc0BieXRlY2FtcC5uZXQ+iQE+BBMBAgAoBQJUab73Ahsj BQkJZgGABgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDDYTHqanMMhMpTB/4xAqtfANpm GYyHTt3z2250jx62GwDfK569dASQpiz1U7AwVvZDfsQSMpsIT/GkaDB5meNZVLYIvfpk1jmi CZ11b0WdERShxQhQsZCBWSIJG4jDfbbmhr212hi1rYzBisLb4+sU/V2Gdd42VUitkPItiv8X ImStSrzWiUAOmeXMEgJA6728sh692oYjsVQVXxRMwcAJSwwnlo32HRuAWK037oJ/04UoZijC chjdp8RIU2ItEjjAWQRcS+KHtj0cJurb+PE6WA8vDU18eLNVUcFRwJToa3dlSLN39R7icGuc SlN+HlCrqgqoHVMjX043wPjf3XEXtLcJTFyYcptyWjCuuQENBFRpvvcBCADhu4eObbfXhhzG 4/onzvW+g5t5k7XJMsLjGl7FAmn26CpSMTkHZCFmUR74Bz7Pr/l4RTRvFt1vnHWX/cKhcwnR g3ZId2a+xljT4pe77gV7+VQwdw5T80aMGe+ASeS0VqdzIyauZl0tzzVqh0/zcNLNE2KTo3el CENru7bOAAJ9nJ1PEVBgpE5P789duErp71HrDGG6SFdk+pFGKK41gJWktt0Tf13gikEiwsL/ xhYzuPqHV3Rj3vsIewbMLkC4Ekrsjqr4gKpTlrLhv9xOG/+SFIpdWy4pKY1dCLsR6lZXtbzE 31jXxuKGRjNlhm9eDfRfrN6LfKq2Z4Y+OZBUMWrbABEBAAGJASUEGAECAA8FAlRpvvcCGwwF CQlmAYAACgkQw2Ex6mpzDIQ0sgf/UI9D2tti9zGNZMgr9dLuMrdi+/olRo0C/PQIsw4NaBCh seXjQgTOdg99WvynZrvtt2eE1YCCnUORMyk1h8dtR3ZmFTMy9DAf7hiY99/avadM+rzoczEV WpZND88rD8JZ1WQKZtTDzH+c2wklkARo4XIu3KDqLEVHhdysN1vACEdHmi1FB3WNSLHF05Mf BUTOsyF3dEP625d03KK/91Sp31F0GJjlSTANxIM05bTcJZsG18auIIevn/19ZXSRdFAC7vy9 ugBfv0HVEC1QsarOCWpqJxQOutZI8I/AywGJetGVkTpKkTRnrIfzoBaFEJFGW96gyLSsWuZs B1zN8/OwHQ== Organization: bytecamp GmbH Message-ID: <92866b76-5f11-2523-cc8f-0d92cc91a50e@bytecamp.net> Date: Fri, 26 Jul 2019 14:52:08 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> Content-Type: text/plain; charset=utf-8 Content-Language: de-DE Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 1FC2F8F01E X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bytecamp.net header.s=20140709 header.b=i7TAPADg; spf=pass (mx1.freebsd.org: domain of rs@bytecamp.net designates 212.204.60.217 as permitted sender) smtp.mailfrom=rs@bytecamp.net X-Spamd-Result: default: False [-2.99 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:212.204.60.0/24]; HAS_ORG_HEADER(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[mail2.bytecamp.net]; DKIM_TRACE(0.00)[bytecamp.net:+]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-0.00)[country: DE(-0.01)]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:12693, ipnet:212.204.32.0/19, country:DE]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[217.60.204.212.list.dnswl.org : 127.0.5.1]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.94)[-0.940,0]; R_DKIM_ALLOW(-0.20)[bytecamp.net:s=20140709]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; DMARC_NA(0.00)[bytecamp.net]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[bytecamp.net.dwl.dnswl.org : 127.0.5.1]; NEURAL_SPAM_SHORT(0.56)[0.563,0]; URIBL_BLOCKED(0.00)[bytecamp.net.multi.uribl.com] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 12:52:19 -0000 Hi, Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions: > Over the past few months i've seen several boxes (4 or 5) become > unresponsive as a result of a Listen queue overflow state. > > Processes stack up, none are killable, all these are within jails and > neither the jail can be stopped nor the server rebooted (without a power > cycle). > > All are on ZFS and are std apache/php/mysql servers with nothing too > exotic. > > All on 12.0-RELEASE, i've only started seeing these issues recently, but > it feels like more and more. > > /var/log/messages shows tyically; > >     kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: > 193 already in queue awaiting acceptance (83 occurrences) > > netstat -Lan  shows > > tcp4 193/0/128                          x.x.x.x.443 > tcp4  193/0/128                          x.x.x.x.80 > > connections cannot be killed with tcpdrop ( except ssh which can!) > > All processes seem to be in Disk State ( many many apache processes but > others getting stuck too) > > www      60089    0.0 0.1  196588   78328  -  DJ   21:07         1:19.54 > /usr/local/sbin/httpd -DNOHTTPACCEPT > .. > > www      93713    0.0 0.0  183576   33164  -  DJ   23:57         0:00.01 > /usr/local/sbin/httpd -DNOHTTPACCEPT > I think, these processes waiting for disk i/o are actually your problem. Since they cannot answer further requests, they run into the listen queue overflow. You should check the processes with procstat: list kernel threads: # procstat -kk list open files: # procstat -f with kind regards, Robert Schulze