From owner-freebsd-questions@freebsd.org Fri Jul 26 13:08:29 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4953DBFB64 for ; Fri, 26 Jul 2019 13:08:29 +0000 (UTC) (envelope-from paul@ifdnrg.com) Received: from outbound.ifdnrg.com (outbound.ifdnrg.com [193.200.98.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "outbound.ifdnrg.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5BB58FADA for ; Fri, 26 Jul 2019 13:08:28 +0000 (UTC) (envelope-from paul@ifdnrg.com) Received: from [192.168.0.210] (38.156-30-62.static.virginmediabusiness.co.uk [62.30.156.38]) (authenticated bits=0) by outbound.ifdnrg.com (8.15.2/8.15.2) with ESMTPSA id x6QD8ODp037114 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 26 Jul 2019 14:08:25 +0100 (BST) (envelope-from paul@ifdnrg.com) X-Authentication-Warning: outbound.ifdnrg.com: Host 38.156-30-62.static.virginmediabusiness.co.uk [62.30.156.38] claimed to be [192.168.0.210] Subject: Re: Help:: Listen queue overflow killing servers To: freebsd-questions@freebsd.org References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> <92866b76-5f11-2523-cc8f-0d92cc91a50e@bytecamp.net> From: Paul Macdonald Message-ID: Date: Fri, 26 Jul 2019 14:08:25 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <92866b76-5f11-2523-cc8f-0d92cc91a50e@bytecamp.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-Rspamd-Queue-Id: B5BB58FADA X-Spamd-Bar: + X-Spamd-Result: default: False [1.14 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:193.200.98.0/23]; TO_DN_NONE(0.00)[]; HAS_XAW(0.00)[]; DKIM_TRACE(0.00)[ifdnrg.com:+]; DMARC_POLICY_ALLOW(0.00)[ifdnrg.com,quarantine]; MX_GOOD(-0.01)[cached: as1.ifdnrg.com]; RCVD_IN_DNSWL_LOW(-0.10)[22.98.200.193.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:20860, ipnet:193.200.98.0/23, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RSPAMD_URIBL(4.50)[ifdnrg.com]; R_DKIM_ALLOW(0.00)[ifdnrg.com:s=ifdnrg-default]; URIBL_BLOCKED(0.00)[ifdnrg.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.40)[-0.404,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; NEURAL_HAM_MEDIUM(-0.80)[-0.800,0]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[ifdnrg.com.dwl.dnswl.org : 127.0.5.1]; BAD_REP_POLICIES(0.10)[]; NEURAL_SPAM_SHORT(0.08)[0.077,0]; IP_SCORE(-1.12)[ipnet: 193.200.98.0/23(-4.77), asn: 20860(-0.75), country: GB(-0.08)]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 13:08:29 -0000 On 26/07/2019 13:52, Robert Schulze wrote: > Hi, > > Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions: > I think, these processes waiting for disk i/o are actually your problem. > Since they cannot answer further requests, they run into the listen > queue overflow. > > You should check the processes with procstat: > > list kernel threads: > # procstat -kk > > list open files: > # procstat -f One of the things we do (whihc may be bad)  is to log to a single file ( e.g all.sites.log, this doesn;t seem to cause problems in general , but i can see how if there's X child processes then they may all need write locks) Is that a really bad idea? ( Often handy to have one file for differnet vhosts, but maybe that needs a rethink) In this case the drive is NVMe, and there's actually  only a handful of sites, (Other servers have several hundreds of sites, much busy but don;t display  the issue) In answer to some of the other suggestions, its not actually under high load ( 5000 lines in the apache log for the whole day), and system has 16C/32T,  128GB RAM ZFS is using a bunch of RAM as i've not limited the ARC, but there's 27GB free currently. I guess i actually have 2 questions 1) why are the queues filling up (i'll  revert to seperate logs to see if that helps, although the issue is sporadic, and first time on this box) 2)  Once the queues are over limit, is this actually unresolvable other than a hard reboot? I've increased the queue size via sysctl kern.ipc.somaxconn to 1024 for now but netstat -Lan still shows tcp4  0/0/128                          x.x.x.x.443 tcp4  0/0/128                          x.x.x.x.80 thanks for assiestence Paul > with kind regards, > Robert Schulze > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > -- ------------------------- Paul Macdonald IFDNRG Ltd Web and video hosting ------------------------- t: 0131 5548070 m: 07970339546 e: paul@ifdnrg.com w: http://www.ifdnrg.com ------------------------- IFDNRG 40 Maritime Street Edinburgh EH6 6SA ---------------------------------------------------- Virtual Servers from £50.00pm High specification Dedicated Servers from £150.00pm ----------------------------------------------------