From owner-freebsd-stable@FreeBSD.ORG Wed Apr 14 05:26:24 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CADE2106564A for ; Wed, 14 Apr 2010 05:26:24 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.westchester.pa.mail.comcast.net (qmta07.westchester.pa.mail.comcast.net [76.96.62.64]) by mx1.freebsd.org (Postfix) with ESMTP id 784388FC2C for ; Wed, 14 Apr 2010 05:26:24 +0000 (UTC) Received: from omta08.westchester.pa.mail.comcast.net ([76.96.62.12]) by qmta07.westchester.pa.mail.comcast.net with comcast id 5HP41e0030Fqzac57HSQVW; Wed, 14 Apr 2010 05:26:24 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta08.westchester.pa.mail.comcast.net with comcast id 5HSP1e0083S48mS3UHSQfd; Wed, 14 Apr 2010 05:26:24 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5931D9B419; Tue, 13 Apr 2010 22:26:22 -0700 (PDT) Date: Tue, 13 Apr 2010 22:26:22 -0700 From: Jeremy Chadwick To: Gabor PALI Message-ID: <20100414052622.GA40757@icarus.home.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-stable@freebsd.org Subject: Re: Strange Swapping Issues(?) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Apr 2010 05:26:25 -0000 On Wed, Apr 14, 2010 at 06:44:58AM +0200, Gabor PALI wrote: > Hello there, > > I am running a FreeBSD/amd64 8-STABLE with GENERIC kernel as of > February 17 on my box (quad core, 2 GB RAM), and recently I spot some > interesting problems in my logs. My machine runs two instances of a > client in two separate chroot environments in parallel with 32-bit and > 64-bit userlands respectively, doing a nightly building and testing. > According to the logs it puts a nice load on my system, though it > still seems to be working fine. > > Except one thing: it produces strange error messages on swap space > without an apparent reason (at least to me): > > xxx# tail -f /var/log/messages > Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(4): failed > Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:44 xxx kernel: > Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(3): failed > Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(3): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(12): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(2): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: pid 7388 (throwto003), uid 1001, was > killed: out of swap space > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(8): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(8): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(12): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(9): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(9): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(3): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(9): failed > Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(3): failed > ^C > xxx# swapinfo -h > Device 1K-blocks Used Avail Capacity > /dev/ad0s1b 4194304 112M 3.9G 3% The swapinfo command you ran was not run at 05:26 in the morning. You should probably set up a small script, run via cronjob, that logs swapinfo -h output to a file somewhere (rotate it if you want via newsyslog.conf). You may have something running on the system that spirals out of control, such as a web board script being pounded to death, or something that's forking excessively. I'd also recommend having the script output "top -b -o res 100", which will give you the top 100 processes on the machine sorted by RSS (non-shared) memory usage. I don't know of a way to show "the amount of swap used by process N, or all processes", since it's transparently handled by the VM. So I'm making the assumption RSS will be large. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |