From owner-freebsd-stable@freebsd.org Thu Aug 6 02:18:31 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD8599B2B09 for ; Thu, 6 Aug 2015 02:18:30 +0000 (UTC) (envelope-from bc979@lafn.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C68E81DE for ; Thu, 6 Aug 2015 02:18:30 +0000 (UTC) (envelope-from bc979@lafn.org) Received: by mailman.ysv.freebsd.org (Postfix) id C377F9B2B08; Thu, 6 Aug 2015 02:18:30 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C22A59B2B07 for ; Thu, 6 Aug 2015 02:18:30 +0000 (UTC) (envelope-from bc979@lafn.org) Received: from zoom.lafn.org (zoom.lafn.org [108.92.93.123]) by mx1.freebsd.org (Postfix) with ESMTP id A33AB1DD for ; Thu, 6 Aug 2015 02:18:30 +0000 (UTC) (envelope-from bc979@lafn.org) Received: from mbook.home (pool-71-178-226-198.washdc.fios.verizon.net [71.178.226.198]) (authenticated bits=0) by zoom.lafn.org (8.14.7/8.14.9) with ESMTP id t762ILR7070581 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Aug 2015 19:18:22 -0700 (PDT) (envelope-from bc979@lafn.org) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Swap Usage From: Doug Hardie In-Reply-To: <10F94D23-E58C-466E-ADCA-5E6670054BD7@lafn.org> Date: Wed, 5 Aug 2015 19:18:21 -0700 Cc: Peter Jeremy , FreeBSD Stable ML Content-Transfer-Encoding: quoted-printable Message-Id: References: <20150730064444.GA88137@server.rulingia.com> <10F94D23-E58C-466E-ADCA-5E6670054BD7@lafn.org> To: Doug Hardie X-Mailer: Apple Mail (2.2102) X-Virus-Scanned: clamav-milter 0.98 at zoom.lafn.org X-Virus-Status: Clean X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 02:18:31 -0000 > On 30 July 2015, at 01:39, Doug Hardie wrote: >=20 >=20 >> On 29 July 2015, at 23:44, Peter Jeremy wrote: >>=20 >> [reformatted] >>=20 >> On 2015-Jul-29 17:41:33 -0700, Doug Hardie wrote: >>> I have several FreeBSD 9.3 systems that are using swap and I can=E2=80= =99t >>> figure out what is doing it. The key system has 6GB swap and >>> currently it has over 2GB in use. >>=20 >> Is the system currently paging (top(1) and "systat -v" will show >> this)? If not, this just means that at some time in the past, the >> system was under memory pressure and paged some process memory out. >> Since then, that memory hasn't been touched so the system hasn't = paged >> it in. >>=20 >>> ps shows only a kernel module >>> [intr] with a W status. >>=20 >> 'W' means the whole process is 'swapped' out - this will only occur >> under severe RAM pressure. Normally, the system will just page out >> inactive parts of a processes address space - and none of the ps = flags >> will show this. >>=20 >>> How do I figure out what that swap space is being used for? >>=20 >> I don't think this can be trivially done. "procstat -v" will show >> the number of resident pages within each swap-backed region, any >> pages in that region that have been touched but are not resident >> are on the swap device but any pages that have never been touched >> aren't counted at all. >=20 > Bingo. procstat shows the problem. The process that I suspected has = a large number of entries like: >=20 > 650 0x834c00000 0x835800000 rw- 0 0 1 0 ---- = sw=20 > 650 0x835800000 0x835c00000 rw- 0 0 1 0 ---- = sw=20 > 650 0x835c00000 0x837c00000 rw- 1 0 1 0 ---- = sw=20 >=20 > I don=E2=80=99t know whats in those areas yet. If I were to kill the = process with SIGABRT would the core dump show those areas? I might be = able to figure out what they are from that. >=20 > Thanks for the pointer. I believe I have found a bug in FreeBSD 9.3 mmap. The following test = program (named test.c) does an mmap to itself and monitors the memory = objects created with procstat. After the mmap, munamp is called and the = objects are checked again. The objects created from the mmap do not get = purged, but remain. Eventually, after enough have been created, they = start becoming type sw and eventually the system runs out of swap. The output: zool# ./test 31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df=20 31459 0x2805f000 0x28069000 rw- 10 0 1 0 C--- df=20 31459 0x28186000 0x281ad000 rw- 13 0 1 0 CN-- df=20 31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df=20 ---------------------------------- 31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df=20 31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df=20 31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df=20 31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df=20 31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df=20 31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df=20 ---------------------------------- 31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df=20 31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df=20 31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df=20 31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df=20 31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df=20 31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df=20 ---------------------------------- 31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df=20 31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df=20 31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df=20 31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df=20 31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df=20 31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df=20 The program: int main (int argc, char *argv[]) { int rc, pid, fd; char *cp; char cmd[1024]; pid =3D getpid (); sprintf (cmd, "procstat -v %d | grep df", pid); fflush (stdout); rc =3D system (cmd); fflush (stdout); printf ("----------------------------------\n"); fd =3D open ("./test.c", O_RDWR); cp =3D mmap (0, 100, PROT_READ|PROT_WRITE, MAP_FILE, fd, 0); if (cp =3D=3D MAP_FAILED) { printf ("mmap error %s\n", strerror(errno)); exit (1); } fflush (stdout); rc =3D system (cmd); fflush (stdout); printf ("----------------------------------\n"); rc =3D munmap (cp, 100); if (rc =3D=3D -1) { printf ("munmap error %s\n", strerror(errno)); exit (1); } fflush (stdout); rc =3D system (cmd); fflush (stdout); printf ("----------------------------------\n"); close (fd); fflush (stdout); rc =3D system (cmd); fflush (stdout); }