From owner-freebsd-stable@freebsd.org Thu Feb 4 15:09:41 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 186B9A99118 for ; Thu, 4 Feb 2016 15:09:41 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [IPv6:2a02:b90:3002:e550::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DEB15BEF for ; Thu, 4 Feb 2016 15:09:40 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from dilbert.london-internal.ingresso.co.uk ([10.64.50.6] helo=dilbert.ingresso.co.uk) by constantine.ingresso.co.uk with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aRLXO-0003Tl-2s for freebsd-stable@FreeBSD.ORG; Thu, 04 Feb 2016 15:09:38 +0000 Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aRLXO-0009T0-0s for freebsd-stable@FreeBSD.ORG; Thu, 04 Feb 2016 15:09:38 +0000 To: freebsd-stable@FreeBSD.ORG Subject: Recent -STABLE crashes out of swap at 3am every day Message-Id: From: Pete French Date: Thu, 04 Feb 2016 15:09:38 +0000 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2016 15:09:41 -0000 I recently updates a system to -STABLE, which ahdnt been updates in 9 months or so. It started locking up at 3am every day. I have updated to -STABLE form this mornign and verified that this still happens. The porobel is the 'periodic daily' process. I have veriied this by running it by hand. What happens is that Swap space starts to fill up. Wathcing in top, however, there are no large processes running. I also start to see these messages during the run... Feb 4 14:57:07 toybox kernel: sonewconn: pcb 0xfffff8000eee6000: Listen queue overflow: 31 already in queue awaiting acceptance (1 occurrences) Feb 4 14:58:07 toybox kernel: sonewconn: pcb 0xfffff8000eee6000: Listen queue overflow: 31 already in queue awaiting acceptance (26 occurrences) Feb 4 14:59:07 toybox kernel: sonewconn: pcb 0xfffff8000eee6000: Listen queue overflow: 31 already in queue awaiting acceptance (193 occurrences) The systems is mainly ZFS, booting from a small UFS partition for /. It has 2GB of RAM only, but the ARC is limited to 512 meg. It primarttily runs exim/clamav/spamd/mailman - these are failmy memory hungry I realise, and the machine usually runs with a few hundred meg swapped out, but iit has been very reliable until the most recent upgrade. I dont knwo preciusely what the periodc scripts do, but I underrstand the use a substantial amount of ;find' and it is this pricess which shows up when runing, usually in the state zio->io_ which is what I would expect. The find process is not using a lot of memory, however. I dont see this on any other achines runnign the most recemt -STABLE, but then again this is my only machine with so little memory in it. What can I do to tracve anmd fix this ? My obvious intiail move is to disable the daily periopdic run, but thats hardly a solution! help? -pete.