From owner-freebsd-questions@freebsd.org Fri Jul 26 21:19:18 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9C888A9A13 for ; Fri, 26 Jul 2019 21:19:18 +0000 (UTC) (envelope-from byrnejb@harte-lyne.ca) Received: from mx32.harte-lyne.ca (mx32.harte-lyne.ca [216.185.71.32]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mx32.harte-lyne.ca", Issuer "CA_HLL_ISSUER_2016" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 36B1683739 for ; Fri, 26 Jul 2019 21:19:08 +0000 (UTC) (envelope-from byrnejb@harte-lyne.ca) Received: from mx32.harte-lyne.ca (unknown [127.0.32.1]) by mx32.harte-lyne.ca (Postfix) with ESMTP id 68D25480C; Fri, 26 Jul 2019 17:18:59 -0400 (EDT) X-Virus-Scanned: amavisd-new at harte-lyne.ca Received: from mx32.harte-lyne.ca ([127.0.32.1]) by mx32.harte-lyne.ca (mx32.harte-lyne.ca [127.0.32.1]) (amavisd-new, port 10024) with ESMTP id 2UWKhR2N7d3X; Fri, 26 Jul 2019 17:18:53 -0400 (EDT) Received: from webmail.harte-lyne.ca (webmail.hamilton.harte-lyne.ca [216.185.71.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx32.harte-lyne.ca (Postfix) with ESMTPSA id BBB484801; Fri, 26 Jul 2019 17:18:52 -0400 (EDT) Received: from 216.185.71.44 (SquirrelMail authenticated user byrnejb_hll) by webmail.harte-lyne.ca with HTTP; Fri, 26 Jul 2019 17:18:53 -0400 Message-ID: <6485e15869f8b205cf36811adaeed0e5.squirrel@webmail.harte-lyne.ca> In-Reply-To: References: Date: Fri, 26 Jul 2019 17:18:53 -0400 Subject: Re: Help:: Listen queue overflow killing servers From: "James B. Byrne" To: "David Christensen" Cc: freebsd-questions@freebsd.org Reply-To: byrnejb@harte-lyne.ca User-Agent: SquirrelMail/1.4.23 [SVN] MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Rspamd-Queue-Id: 36B1683739 X-Spamd-Bar: -------- X-Spamd-Result: default: False [-8.47 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[byrnejb@harte-lyne.ca]; RBL_COMPOSITE_RCVD_IN_DNSWL_MED_DWL_DNSWL_LOW(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:216.185.71.0/26]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DKIM_TRACE(0.00)[harte-lyne.ca:+]; RCPT_COUNT_TWO(0.00)[2]; HAS_X_PRIO_THREE(0.00)[3]; RCVD_IN_DNSWL_MED(-0.20)[32.71.185.216.list.dnswl.org : 127.0.4.2]; DMARC_POLICY_ALLOW(-0.50)[harte-lyne.ca,quarantine]; MX_GOOD(-0.01)[mx32.harte-lyne.ca,mx31.harte-lyne.ca]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:12021, ipnet:216.185.64.0/20, country:CA]; IP_SCORE(-3.77)[ip: (-9.90), ipnet: 216.185.64.0/20(-4.93), asn: 12021(-3.94), country: CA(-0.09)]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[harte-lyne.ca:s=dkim_hll]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; URIBL_BLOCKED(0.00)[harte-lyne.ca.multi.uribl.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_LOW(0.00)[harte-lyne.ca.dwl.dnswl.org : 127.0.4.1]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.984,0] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 21:19:18 -0000 >>>>> On 7/26/19 4:58 AM, Paul Macdonald via freebsd-questions wrote: >>>>>> Over the past few months i've seen several boxes (4 or 5) become >>>>>> unresponsive as a result of a Listen queue overflow state. Since upgrading our hosts to 12.0 we have experienced many 'lockouts' of both bhyve vm guests and jails, all running on zfs. There are known issues with Bhyve guests getting into a deadlock state waiting on zio or encountering memory exhaustion in releases after FreeBSD-11.1. This has multiple causes: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 We have worked around this by restricting max ARC to the net of system memory minus all vm allocations minus 4GB. We also noted that as the capacity of zfs pools approached 80% the deadlock is encountered more frequently. -- *** e-Mail is NOT a SECURE channel *** Do NOT transmit sensitive data via e-Mail Do NOT open attachments nor follow links sent by e-Mail James B. Byrne mailto:ByrneJB@Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3