Date: Fri, 26 Jul 2019 14:08:25 +0100 From: Paul Macdonald <paul@ifdnrg.com> To: freebsd-questions@freebsd.org Subject: Re: Help:: Listen queue overflow killing servers Message-ID: <a4675162-4241-2145-3380-c12253032da1@ifdnrg.com> In-Reply-To: <92866b76-5f11-2523-cc8f-0d92cc91a50e@bytecamp.net> References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> <92866b76-5f11-2523-cc8f-0d92cc91a50e@bytecamp.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 26/07/2019 13:52, Robert Schulze wrote: > Hi, > > Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions: > I think, these processes waiting for disk i/o are actually your problem. > Since they cannot answer further requests, they run into the listen > queue overflow. > > You should check the processes with procstat: > > list kernel threads: > # procstat -kk <PID> > > list open files: > # procstat -f <PID> One of the things we do (whihc may be bad) is to log to a single file ( e.g all.sites.log, this doesn;t seem to cause problems in general , but i can see how if there's X child processes then they may all need write locks) Is that a really bad idea? ( Often handy to have one file for differnet vhosts, but maybe that needs a rethink) In this case the drive is NVMe, and there's actually only a handful of sites, (Other servers have several hundreds of sites, much busy but don;t display the issue) In answer to some of the other suggestions, its not actually under high load ( 5000 lines in the apache log for the whole day), and system has 16C/32T, 128GB RAM ZFS is using a bunch of RAM as i've not limited the ARC, but there's 27GB free currently. I guess i actually have 2 questions 1) why are the queues filling up (i'll revert to seperate logs to see if that helps, although the issue is sporadic, and first time on this box) 2) Once the queues are over limit, is this actually unresolvable other than a hard reboot? I've increased the queue size via sysctl kern.ipc.somaxconn to 1024 for now but netstat -Lan still shows tcp4 0/0/128 x.x.x.x.443 tcp4 0/0/128 x.x.x.x.80 thanks for assiestence Paul > with kind regards, > Robert Schulze > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > -- ------------------------- Paul Macdonald IFDNRG Ltd Web and video hosting ------------------------- t: 0131 5548070 m: 07970339546 e: paul@ifdnrg.com w: http://www.ifdnrg.com ------------------------- IFDNRG 40 Maritime Street Edinburgh EH6 6SA ---------------------------------------------------- Virtual Servers from £50.00pm High specification Dedicated Servers from £150.00pm ----------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a4675162-4241-2145-3380-c12253032da1>