Date: Thu, 16 Jul 2015 21:19:13 +0300 From: Dmitry Sivachenko <trtrmitya@gmail.com> To: hackers@freebsd.org Subject: Re: Strange memory management with mmap() Message-ID: <3434ED75-7994-4E9E-9B06-FACCD7DC90FF@gmail.com> In-Reply-To: <FDB6E0F9-A3FF-4194-83C1-A3121CBAE407@gmail.com> References: <FDB6E0F9-A3FF-4194-83C1-A3121CBAE407@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 16 =D0=B8=D1=8E=D0=BB=D1=8F 2015 =D0=B3., at 18:42, Dmitry =
Sivachenko <trtrmitya@gmail.com> wrote:
>=20
> Hello!
>=20
> I am using FreeBSD-10-stable and writing a program that uses large =
data file via mmap() in read only mode.
> To be specific, I have 256GB RAM machine and typical size of data file =
is ~160GB (more than 1/2 of RAM and less that the whole RAM).
> There is no other programs running during the test.
>=20
> Consider the following use case: I have two files on disk. I mmap() =
the first one and prefetch data to RAM (touch every page of the file).
> After that I expect all data to be cached in RAM and subsequent access =
will be fast.
>=20
> Next I do munmap() on the first file, mmap() the second one and do the =
same test: prefetch data and expect it to be cached in RAM (and some of =
the pages belonging to the first file to be purged out, because =
size_of(file1)+size_of(file2) > size_of(RAM).
>=20
> Please find my test program attached.
>=20
> I run the program with 2 files provided via command line (both about =
160GB).
> What I observe in real is:
> -- before I run the program all RAM is in FREE state as reported by =
top(1).
> -- after first prefetch() of the first file, all it's data goes to =
"Cache" state, RES column of the process remains the same (small)
> -- second prefetch() works fast as expected, memory goes from Cache to =
Active state, RES column of the process grows up to match file size =
(SIZE=3D=3DRES now)
> -- now first prefetch() for second file starts: the remaining Free =
memory goes to Cache state, Active size still equals to first file size.
> -- second prefetch() for second file works as slow as first one, like =
if nothing was cached in memory during the first prefetch() run, RES =
column does not change.
>=20
>=20
> Here is the output:
> % /tmp/a.out file1.dat file2.dat
> file1.dat... First prefault time: 1235.747351 seconds
> Second prefault time: 74.893323 seconds
> Ok.
> file2.dat... First prefault time: 1316.405527 seconds
> Second prefault time: 1311.491842 seconds
> Ok.
>=20
And one more strange observation I made:=20
if I add=20
if (madvise(p, st.st_size, MADV_RANDOM) !=3D 0)
err(1, "madvise");
if (madvise(p, st.st_size, MADV_WILLNEED) !=3D 0)
err(1, "madvise");
after mmap() call in my test program, these times change as follows:
% /tmp/a.out file1.dat file2.dat
file1.dat... First prefault time: 2706.270454 seconds
Second prefault time: 3.812012 seconds
Ok.
file2.dat... First prefault time: 4089.096647 seconds
Second prefault time: 1132.210888 seconds
Ok.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3434ED75-7994-4E9E-9B06-FACCD7DC90FF>