Date: Wed, 5 Aug 2015 19:18:21 -0700 From: Doug Hardie <bc979@lafn.org> To: Doug Hardie <bc979@lafn.org> Cc: Peter Jeremy <peter@rulingia.com>, FreeBSD Stable ML <stable@freebsd.org> Subject: Re: Swap Usage Message-ID: <BCB80170-3E1E-4861-B159-CD52472FB475@lafn.org> In-Reply-To: <10F94D23-E58C-466E-ADCA-5E6670054BD7@lafn.org> References: <BCA67F7E-676A-4226-83A0-84229948895E@lafn.org> <20150730064444.GA88137@server.rulingia.com> <10F94D23-E58C-466E-ADCA-5E6670054BD7@lafn.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 30 July 2015, at 01:39, Doug Hardie <bc979@lafn.org> wrote:
>=20
>=20
>> On 29 July 2015, at 23:44, Peter Jeremy <peter@rulingia.com> wrote:
>>=20
>> [reformatted]
>>=20
>> On 2015-Jul-29 17:41:33 -0700, Doug Hardie <bc979@lafn.org> wrote:
>>> I have several FreeBSD 9.3 systems that are using swap and I can=E2=80=
=99t
>>> figure out what is doing it. The key system has 6GB swap and
>>> currently it has over 2GB in use.
>>=20
>> Is the system currently paging (top(1) and "systat -v" will show
>> this)? If not, this just means that at some time in the past, the
>> system was under memory pressure and paged some process memory out.
>> Since then, that memory hasn't been touched so the system hasn't =
paged
>> it in.
>>=20
>>> ps shows only a kernel module
>>> [intr] with a W status.
>>=20
>> 'W' means the whole process is 'swapped' out - this will only occur
>> under severe RAM pressure. Normally, the system will just page out
>> inactive parts of a processes address space - and none of the ps =
flags
>> will show this.
>>=20
>>> How do I figure out what that swap space is being used for?
>>=20
>> I don't think this can be trivially done. "procstat -v" will show
>> the number of resident pages within each swap-backed region, any
>> pages in that region that have been touched but are not resident
>> are on the swap device but any pages that have never been touched
>> aren't counted at all.
>=20
> Bingo. procstat shows the problem. The process that I suspected has =
a large number of entries like:
>=20
> 650 0x834c00000 0x835800000 rw- 0 0 1 0 ---- =
sw=20
> 650 0x835800000 0x835c00000 rw- 0 0 1 0 ---- =
sw=20
> 650 0x835c00000 0x837c00000 rw- 1 0 1 0 ---- =
sw=20
>=20
> I don=E2=80=99t know whats in those areas yet. If I were to kill the =
process with SIGABRT would the core dump show those areas? I might be =
able to figure out what they are from that.
>=20
> Thanks for the pointer.
I believe I have found a bug in FreeBSD 9.3 mmap. The following test =
program (named test.c) does an mmap to itself and monitors the memory =
objects created with procstat. After the mmap, munamp is called and the =
objects are checked again. The objects created from the mmap do not get =
purged, but remain. Eventually, after enough have been created, they =
start becoming type sw and eventually the system runs out of swap.
The output:
zool# ./test
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df=20
31459 0x2805f000 0x28069000 rw- 10 0 1 0 C--- df=20
31459 0x28186000 0x281ad000 rw- 13 0 1 0 CN-- df=20
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df=20
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df=20
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df=20
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df=20
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df=20
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df=20
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df=20
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df=20
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df=20
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df=20
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df=20
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df=20
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df=20
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df=20
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df=20
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df=20
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df=20
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df=20
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df=20
The program:
int main (int argc, char *argv[])
{
int rc, pid, fd;
char *cp;
char cmd[1024];
pid =3D getpid ();
sprintf (cmd, "procstat -v %d | grep df", pid);
fflush (stdout);
rc =3D system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
fd =3D open ("./test.c", O_RDWR);
cp =3D mmap (0, 100, PROT_READ|PROT_WRITE, MAP_FILE, fd, 0);
if (cp =3D=3D MAP_FAILED)
{
printf ("mmap error %s\n", strerror(errno));
exit (1);
}
fflush (stdout);
rc =3D system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
rc =3D munmap (cp, 100);
if (rc =3D=3D -1)
{
printf ("munmap error %s\n", strerror(errno));
exit (1);
}
fflush (stdout);
rc =3D system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
close (fd);
fflush (stdout);
rc =3D system (cmd);
fflush (stdout);
}
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BCB80170-3E1E-4861-B159-CD52472FB475>