From owner-freebsd-questions@freebsd.org Fri Jul 26 12:41:32 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E6F87BF172 for ; Fri, 26 Jul 2019 12:41:32 +0000 (UTC) (envelope-from mafsys1234@gmail.com) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 297178E6FB for ; Fri, 26 Jul 2019 12:41:32 +0000 (UTC) (envelope-from mafsys1234@gmail.com) Received: by mail-pg1-x535.google.com with SMTP id l21so24740479pgm.3 for ; Fri, 26 Jul 2019 05:41:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=xNCpZFYw7fK6+PU1XiI4m5UBi1853NRV+LeKTzXKZs8=; b=WHslzHmVFSJJfKyYkuQXijnyhZh6z0Co3Dg5EPPMQW/HgT/M078MZ2k4JeSv/um5JH yqwo+lgqM+/j08UzqEoIfP6llZw1PjO9KUaSl9NLItxNUkVlynF+XfGDTtcnXsOq07cl jS9L6rdFbOjzHI7cBdsRdjaBQJY/sAptHJSv6ys199boZRI/V+O86ZXsvVSjjztYPjqL H1tAzEuz764Brqrl2dwf5gorqGa9RDu/O2kNZOooo23WB87L5g9Tognn7OYJX//U/Ww1 ssx/Buyex8lwpZeGNdXQjCoDSwFDWkFmCRERNorDjVVo+figluKKpGO63/vaCHX99BRV qTSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=xNCpZFYw7fK6+PU1XiI4m5UBi1853NRV+LeKTzXKZs8=; b=bxMZ9kCvXVtF7+AmQfuw8GDuYmOs5QRdegflwG3o5v9UgrQiQRzKXypEo7K1QhZH5o gFTF9qwAolGANzYcEY+4e3MF/cLJEgNbh0aAM1ksz8gjaOn5JDGACopyAApca24GiaEZ rgmLhR8xtWc5N494x5v1GDddTBn1pav22w0jmqgxYxbvLutpqyvkrDYG3/IagMw84YLm K7TGPaGz/wgvrvDAyCEhCFbV8n6vjD4qSEPa7VWpBeylTmLr+i5L+sEhQYb3+P7NrK+u 7fsyeYWDKeLttWqf5+k+WtkU96o+7Iqgeie72WoM+a6BY60LU8iFzOYFnJd/bRzvGm3e mHhA== X-Gm-Message-State: APjAAAUsqh76O8ZJtXeB+MMir6owwEz2Puo7mu1wYqDi3RciKhGHRmTd 4RpklEMMXdSaAHly7PFoI1TGPfoiAiM= X-Google-Smtp-Source: APXvYqyZ2UGbIgcq8sDQFcIP0TqV9IMjabPHeFhQ/RofSMKh9O0eugteg4j0tuQeyA6SYWv+SVbpEQ== X-Received: by 2002:a65:40cc:: with SMTP id u12mr16215654pgp.372.1564144890716; Fri, 26 Jul 2019 05:41:30 -0700 (PDT) Received: from [192.168.1.10] ([210.9.159.55]) by smtp.gmail.com with ESMTPSA id 3sm56321850pfg.186.2019.07.26.05.41.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Jul 2019 05:41:30 -0700 (PDT) Subject: Re: Help:: Listen queue overflow killing servers To: freebsd-questions@freebsd.org References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> From: MJ Message-ID: <907bd63c-7487-0b01-b202-7c67b8c0d07f@gmail.com> Date: Fri, 26 Jul 2019 22:41:01 +1000 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 297178E6FB X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=WHslzHmV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mafsys1234@gmail.com designates 2607:f8b0:4864:20::535 as permitted sender) smtp.mailfrom=mafsys1234@gmail.com X-Spamd-Result: default: False [0.05 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.33)[-0.335,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; RSPAMD_URIBL(4.50)[ifdnrg.com]; R_DKIM_ALLOW(0.00)[gmail.com:s=20161025]; URIBL_BLOCKED(0.00)[ifdnrg.com.multi.uribl.com]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.42)[-0.422,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; NEURAL_HAM_MEDIUM(-0.68)[-0.679,0]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; IP_SCORE(-3.01)[ip: (-9.46), ipnet: 2607:f8b0::/32(-3.09), asn: 15169(-2.44), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[5.3.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 12:41:33 -0000 I'd also look at why it's disk bound. I don't know the ops knowledge of dtrace but running truss on one of the stuck processes might given him a clue as to why these processes are bottlenecking the system. You could probably up kern.ipc.soacceptqueue for some temporary relief but I'd suggest that would just mask the problem. Cheers Mark On 26/07/2019 10:28 pm, Igor V. Ruzanov wrote: > You can increase TCP queue at system level: > sysctl kern.ipc.somaxconn=65535 (its maximum value). But often its not > enough because of application you use. So dig into application settings to > increase the queue of TCP sockets to be handled in your system. > > > On Fri, 26 Jul 2019, Paul Macdonald via freebsd-questions wrote: > > | > |Hi, > | > |Over the past few months i've seen several boxes (4 or 5) become unresponsive > |as a result of a Listen queue overflow state. > | > |Processes stack up, none are killable, all these are within jails and neither > |the jail can be stopped nor the server rebooted (without a power cycle). > | > |All are on ZFS and are std apache/php/mysql servers with nothing too exotic. > | > |All on 12.0-RELEASE, i've only started seeing these issues recently, but it > |feels like more and more. > | > |/var/log/messages shows tyically; > | > |    kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: 193 > |already in queue awaiting acceptance (83 occurrences) > | > |netstat -Lan  shows > | > |tcp4 193/0/128                          x.x.x.x.443 > |tcp4  193/0/128                          x.x.x.x.80 > | > |connections cannot be killed with tcpdrop ( except ssh which can!) > | > |All processes seem to be in Disk State ( many many apache processes but others > |getting stuck too) > | > |www      60089    0.0 0.1  196588   78328  -  DJ   21:07         1:19.54 > |/usr/local/sbin/httpd -DNOHTTPACCEPT > |.. > | > |www      93713    0.0 0.0  183576   33164  -  DJ   23:57         0:00.01 > |/usr/local/sbin/httpd -DNOHTTPACCEPT > | > |but no zombies.. > | > |last pid: 24773;  load averages:  0.00,  0.00, 0.00                       up > |52+11:41:09  11:48:02 > |918 processes: 1 running, 917 sleeping > |CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle > |Mem: 107M Active, 3729M Inact, 93G Wired, 27G Free > |ARC: 79G Total, 54G MFU, 23G MRU, 243M Anon, 710M Header, 1615M Other > |     73G Compressed, 191G Uncompressed, 2.60:1 Ratio > |Swap: 4096M Total, 4096M Free > | > | > |I'd appreciate any advice as at present it looks like my only option is to hard > |power cycle these > | > |many thanks > | > |Paul. > | > | > | > |-- > |------------------------- > |Paul Macdonald > |IFDNRG Ltd > |Web and video hosting > |------------------------- > |t: 0131 5548070 > |m: 07970339546 > |e: paul@ifdnrg.com > |w: http://www.ifdnrg.com > |------------------------- > |IFDNRG > |40 Maritime Street > |Edinburgh > |EH6 6SA > |---------------------------------------------------- > | > |Virtual Servers from £50.00pm > |High specification Dedicated Servers from £150.00pm > | > |---------------------------------------------------- > | > |_______________________________________________ > |freebsd-questions@freebsd.org mailing list > |https://lists.freebsd.org/mailman/listinfo/freebsd-questions > |To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > | > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"