From owner-freebsd-stable@FreeBSD.ORG Sat Nov 27 15:26:28 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2BBC106564A for ; Sat, 27 Nov 2010 15:26:27 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta02.westchester.pa.mail.comcast.net (qmta02.westchester.pa.mail.comcast.net [76.96.62.24]) by mx1.freebsd.org (Postfix) with ESMTP id 8D46B8FC1A for ; Sat, 27 Nov 2010 15:26:27 +0000 (UTC) Received: from omta22.westchester.pa.mail.comcast.net ([76.96.62.73]) by qmta02.westchester.pa.mail.comcast.net with comcast id cEi41f0041ap0As52FST5E; Sat, 27 Nov 2010 15:26:27 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta22.westchester.pa.mail.comcast.net with comcast id cFSS1f0023LrwQ23iFSSw0; Sat, 27 Nov 2010 15:26:27 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id A525F9B422; Sat, 27 Nov 2010 07:26:24 -0800 (PST) Date: Sat, 27 Nov 2010 07:26:24 -0800 From: Jeremy Chadwick To: Ian Smith Message-ID: <20101127152624.GA2077@icarus.home.lan> References: <1DDFC51B19BA4655A2B8AFE4EC536337@jarasc430> <20101127031259.7CD631CC0C@ptavv.es.net> <20101127121743.GA97603@icarus.home.lan> <20101128001107.L47536@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20101128001107.L47536@sola.nimnet.asn.au> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: memory leak and swapfile X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Nov 2010 15:26:28 -0000 On Sun, Nov 28, 2010 at 01:13:37AM +1100, Ian Smith wrote: > On Sat, 27 Nov 2010, Jeremy Chadwick wrote: > > On Fri, Nov 26, 2010 at 07:12:59PM -0800, Kevin Oberman wrote: > > > > From: "Jack Raats" > > > > Date: Fri, 26 Nov 2010 19:17:05 +0100 > > > > Sender: owner-freebsd-stable@freebsd.org > > > > > > > > It looks like that there may be a memory leak of my swap space with one of > > > > the processes that is running. > > > > Big question: How can I determine which process is responsible. > > > > > > > > Any suggestions? > > > > > > ps -aux Look for processes with large values in the VSZ column. > > > > > > I'm sure that there are other ways to see this, but that's an easy > > > one. You can, of course, pipe the output to sort and use the -k 5 -n > > > options. > > > > I believe he should be looking for a process that has a large value in > > RSS ("RES" in top), not VSZ ("SIZE" in top). > > Not necessarily. More noticeable in physical-memory constrained systems > (try 160MB!), processes that leak memory badly (say mozilla) push leaked > but quiescent memory out to swap, where it lurks forever, growing with > every page of eg crappy javascript, while RSS stays more constrained. > > % ps aux | sort -rnk5 | head -2 > smithi 11451 0.0 27.3 119008 42276 ?? S 11Nov10 336:47.74 /usr/local/lib/mozilla/mozilla-bin > root 45236 1.9 7.8 45328 12132 v7 R 15Aug10 3261:37.36 X :0 -auth /home/smithi/.serverau > > At least 35MB of moz's VSZ is leaked and will disappear once restarted, > plus another 15MB or so of VSZ X is hanging onto on mozilla's account. > > Small system, small example; big-iron folks miss out on half the fun :) Here's an example (and not really from a "big-iron" system), though swap isn't used in this scenario. # pstat -s Device 1K-blocks Used Avail Capacity /dev/ada0s1b 16777216 0 16777216 0% # top -b | egrep 'COMMAND|mysqld' PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1199 mysql 10 44 0 801M 271M sigwai 1 16:28 0.00% mysqld # ps -auxw | grep 1199 mysql 1199 0.0 3.3 820544 277924 ?? S 19Oct10 123:00.88 /usr/local/libexec/mysqld --defaults-extra-file=/storage/mysql/my.c # procstat -v 1199 | sort -k5 -n | tail -5 1199 0x90600000 0x90a00000 rw- 612 0 1 0 -- df 1199 0x400000 0x8f0000 r-x 1206 1564 2 1 CN vn /usr/local/libexec/mysqld 1199 0x90c00000 0x91400000 rw- 1276 0 2 0 -- df 1199 0x8fe00000 0x90600000 rw- 2044 0 1 0 -- df 1199 0x61600000 0x8fe00000 rw- 60285 0 2 0 -- df # procstat -v 1199 | sort -k6 -n | tail -5 1199 0x7fffffbff000 0x7fffffc00000 --- 0 0 0 0 -- -- 1199 0x7ffffffe0000 0x800000000000 rwx 15 0 1 0 -- df 1199 0x61190000 0x6119c000 r-x 10 14 2 1 CN vn /lib/libgcc_s.so.1 1199 0x60e68000 0x60f3c000 r-x 126 252 2 1 CN vn /usr/lib/libstdc++.so.6 1199 0x400000 0x8f0000 r-x 1206 1564 2 1 CN vn /usr/local/libexec/mysqld # procstat -v 1199 | awk '/1199/ { res += $5; pres += $6 } END { print res " " pres }' 69267 1830 Now what's important to take into mind is that the counts shown in columns 5 and 6 are page counts, not actual memory usage. I'm making the assumption that page size is hw.pagesize even though the system shows 2MB pages available in hw.pagesizes. So: 69267 * 4096 = 283717632 bytes 1830 * 4096 = 7495680 bytes This more or less matches what's in RSS/RES (the ~10MByte difference probably has to do with scaling of bytes -> megabytes). -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |