From owner-freebsd-questions@freebsd.org Fri Jul 26 13:19:45 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E083AA1195 for ; Fri, 26 Jul 2019 13:19:45 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: from mxout01.bytecamp.net (mxout01.bytecamp.net [212.204.60.217]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 69BBC684BE for ; Fri, 26 Jul 2019 13:19:45 +0000 (UTC) (envelope-from rs@bytecamp.net) Received: by mxout01.bytecamp.net (Postfix, from userid 1001) id 8E81C55E98; Fri, 26 Jul 2019 15:19:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bytecamp.net; h=subject:to:references:from:message-id:date:mime-version:in-reply-to:content-type:content-transfer-encoding; s=20140709; bh=rPBVD7kgTzQkCPJoJjRNcjUoiZw=; b=dvgIptQ7NB3V3GD0DT9mDmbPsJMd3jRRdjQecYBMzj6n/ET4k+6N0pAaftTUYRKi4n4EM9U3KEPpqUH+az6YFHGRK8lF0rC5Dc7HOoF7KOtODgXgq9Jk6Tk/fNytqGqTkRhYyHFzBnbrNKd9wCQv1TRKihkNyjiX3NOek2TCHHM= Received: from mail.bytecamp.net (mail.bytecamp.net [212.204.60.9]) by mxout01.bytecamp.net (Postfix) with ESMTP id 582EA55E95 for ; Fri, 26 Jul 2019 15:19:43 +0200 (CEST) Received: (qmail 74989 invoked from network); 26 Jul 2019 15:19:43 +0200 Received: from unknown (HELO ?192.168.3.59?) (rs%bytecamp.net@80.84.212.123) by mail.bytecamp.net with ESMTPS (AES128-SHA encrypted); 26 Jul 2019 15:19:43 +0200 Subject: Re: Help:: Listen queue overflow killing servers To: freebsd-questions@freebsd.org References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> <92866b76-5f11-2523-cc8f-0d92cc91a50e@bytecamp.net> From: Robert Schulze Openpgp: preference=signencrypt Autocrypt: addr=rs@bytecamp.net; prefer-encrypt=mutual; keydata= mQENBFRpvvcBCACiXR5ENQUFzwmQCyyQmpG3ppKXS5GamZHZSl6dr+yeTRHW5XJ+46ltis64 U/i1V0DhBD9kHjAvwZvWO+BXVj1kf1GFf286WoUnFvRK6oLRtaqdZDDu+AA3EcvvryRzlkE9 uSBXkg+4ObhxhaLhjWowDBhmAyi3QOAtPpw9tme4c0ktoKMMlkryTAwPXw7sJi7TGJXv088T z2C2eMBKNKwyCWmGsP+0UoZRxO0A7qCU1kIGvsf7N9sYFD3OI9NaYZHvN8g7x7NTInz77a2o /u1EFYcQ6UbphP8AhUX9phvQLsZr0w0eCFQ+kJqtHWdesRckQcpDo8eXDQOvZH5vxErzABEB AAG0IFJvYmVydCBTY2h1bHplIDxyc0BieXRlY2FtcC5uZXQ+iQE+BBMBAgAoBQJUab73Ahsj BQkJZgGABgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDDYTHqanMMhMpTB/4xAqtfANpm GYyHTt3z2250jx62GwDfK569dASQpiz1U7AwVvZDfsQSMpsIT/GkaDB5meNZVLYIvfpk1jmi CZ11b0WdERShxQhQsZCBWSIJG4jDfbbmhr212hi1rYzBisLb4+sU/V2Gdd42VUitkPItiv8X ImStSrzWiUAOmeXMEgJA6728sh692oYjsVQVXxRMwcAJSwwnlo32HRuAWK037oJ/04UoZijC chjdp8RIU2ItEjjAWQRcS+KHtj0cJurb+PE6WA8vDU18eLNVUcFRwJToa3dlSLN39R7icGuc SlN+HlCrqgqoHVMjX043wPjf3XEXtLcJTFyYcptyWjCuuQENBFRpvvcBCADhu4eObbfXhhzG 4/onzvW+g5t5k7XJMsLjGl7FAmn26CpSMTkHZCFmUR74Bz7Pr/l4RTRvFt1vnHWX/cKhcwnR g3ZId2a+xljT4pe77gV7+VQwdw5T80aMGe+ASeS0VqdzIyauZl0tzzVqh0/zcNLNE2KTo3el CENru7bOAAJ9nJ1PEVBgpE5P789duErp71HrDGG6SFdk+pFGKK41gJWktt0Tf13gikEiwsL/ xhYzuPqHV3Rj3vsIewbMLkC4Ekrsjqr4gKpTlrLhv9xOG/+SFIpdWy4pKY1dCLsR6lZXtbzE 31jXxuKGRjNlhm9eDfRfrN6LfKq2Z4Y+OZBUMWrbABEBAAGJASUEGAECAA8FAlRpvvcCGwwF CQlmAYAACgkQw2Ex6mpzDIQ0sgf/UI9D2tti9zGNZMgr9dLuMrdi+/olRo0C/PQIsw4NaBCh seXjQgTOdg99WvynZrvtt2eE1YCCnUORMyk1h8dtR3ZmFTMy9DAf7hiY99/avadM+rzoczEV WpZND88rD8JZ1WQKZtTDzH+c2wklkARo4XIu3KDqLEVHhdysN1vACEdHmi1FB3WNSLHF05Mf BUTOsyF3dEP625d03KK/91Sp31F0GJjlSTANxIM05bTcJZsG18auIIevn/19ZXSRdFAC7vy9 ugBfv0HVEC1QsarOCWpqJxQOutZI8I/AywGJetGVkTpKkTRnrIfzoBaFEJFGW96gyLSsWuZs B1zN8/OwHQ== Organization: bytecamp GmbH Message-ID: Date: Fri, 26 Jul 2019 15:19:42 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: de-DE Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 69BBC684BE X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bytecamp.net header.s=20140709 header.b=dvgIptQ7; spf=pass (mx1.freebsd.org: domain of rs@bytecamp.net designates 212.204.60.217 as permitted sender) smtp.mailfrom=rs@bytecamp.net X-Spamd-Result: default: False [-7.99 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:212.204.60.0/24]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; DKIM_TRACE(0.00)[bytecamp.net:+]; MX_GOOD(-0.01)[cached: mail2.bytecamp.net]; NEURAL_HAM_SHORT(-0.87)[-0.873,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-3.51)[ip: (-9.23), ipnet: 212.204.32.0/19(-4.62), asn: 12693(-3.69), country: DE(-0.01)]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:12693, ipnet:212.204.32.0/19, country:DE]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[217.60.204.212.list.dnswl.org : 127.0.5.1]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bytecamp.net:s=20140709]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; DMARC_NA(0.00)[bytecamp.net]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[bytecamp.net.dwl.dnswl.org : 127.0.5.1]; URIBL_BLOCKED(0.00)[bytecamp.net.multi.uribl.com] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 13:19:45 -0000 Hi, Am 26.07.19 um 15:08 schrieb Paul Macdonald via freebsd-questions: > 1) why are the queues filling up (i'll  revert to seperate logs to see > if that helps, although the issue is sporadic, and first time on this box) like I already mentioned: if a webserver process is stuck in disk-wait, this process will probably not answer any further requests, so the listen backlog will fill up. > 2)  Once the queues are over limit, is this actually unresolvable other > than a hard reboot? again: your problem are the webserver processes in disk-wait. This process state is uninterruptible, processes in this state cannot be killed. A process simply reaching somaxcon will not interfere with reboot/kill. > I've increased the queue size via > > sysctl kern.ipc.somaxconn > > to 1024 for now You will not solve your problem by increasing somaxconn. You have to figure out why these processes are stuck in disk i/o (see previous hints about procstat). regards Robert Schulze