Date: Wed, 08 Oct 2014 12:03:21 -0400 From: Richard Yao <ryao@gentoo.org> To: Dmitry Sivachenko <trtrmitya@gmail.com>, "hackers@freebsd.org" <hackers@freebsd.org> Subject: Re: mmap() question Message-ID: <54356049.9040009@gentoo.org> In-Reply-To: <95E0B821-BF9B-4EBF-A1E5-1DDCBB1C3D1B@gmail.com> References: <95E0B821-BF9B-4EBF-A1E5-1DDCBB1C3D1B@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] It might be worthwhile to use madvise() in your code: http://man7.org/linux/man-pages/man2/madvise.2.html On 10/09/2013 07:42 AM, Dmitry Sivachenko wrote: > Hello! > > I have a program which mmap()s a lot of large files (total size more that RAM and I have no swap), but it needs only small parts of that files at a time. > > My understanding is that when using mmap when I access some memory region OS reads the relevant portion of that file from disk and caches the result in memory. If there is no free memory, OS will purge previously read part of mmap'ed file to free memory for the new chunk. > > But this is not the case. I use the following simple program which gets list of files as command line arguments, mmap()s them all and then selects random file and random 1K parts of that file and computes a XOR of bytes from that region. > After some time the program dies: > pid 63251 (a.out), uid 1232, was killed: out of swap space > > It seems I incorrectly understand how mmap() works, can you please clarify what's going wrong? > > I expect that program to run indefinitely, purging some regions out of RAM and reading the relevant parts of files. > > Thanks! > > #include <err.h> > #include <fcntl.h> > #include <math.h> > #include <stdio.h> > #include <stdlib.h> > #include <sys/mman.h> > #include <sys/stat.h> > #include <sys/types.h> > #include <unistd.h> > > struct f_data { > char *beg; > off_t size; > }; > > int > main(int argc, char* argv[]) { > if (argc < 2) { > fprintf(stderr, "Usage: %s <file> ...\n", argv[0]); > exit(0); > } > int i, j, fd; > struct stat st; > struct f_data FILES[500]; > int NUM_FILES; > void *p; > NUM_FILES = argc - 1; > for (i=1; i < argc; i++) { > printf("%s... ", argv[i]); > if ((fd = open(argv[i], O_RDONLY)) < 0) > errx(1, "open"); > if (fstat(fd, &st) != 0) > errx(1, "fstat"); > if ((p = mmap(NULL, st.st_size, PROT_READ, MAP_NOCORE, fd, 0)) == MAP_FAILED) > errx(1, "mmap"); > FILES[i-1].beg = (char*)p; > FILES[i-1].size = st.st_size; > if (msync(p, st.st_size, MS_INVALIDATE) != 0) > errx(1, "msync"); > printf("Ok.\n"); > } > char chk = 0; > while(1) { > int rf = floor((double)random() / 2147483647 * NUM_FILES); > off_t offs = floor((double)random() / 2147483647 * (FILES[rf].size - 1024)); > for (j=0; j<1024; j++) > chk ^= *(FILES[rf].beg + offs + j); > } > return 0; > } > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJUNWBOAAoJECDuEZm+6ExkNX0P/iyUJNHnAXcztp3DoMxlDkKX tatK+LBM+QL/+hFcIE4t4UDdSZ3g5+JN4lCCShuFZHH8UtJcU8BwLmAJhlsBlE2q d3o40SO96eNkelw1ymgBNdzFhb44/PjAy9DkWDMV1tlWX0oi2BTqmeEyf908rASJ odWh42+YwiLRPo+iTAqebIxix4jP9P9mXWEXjbHsNZhrpI6Oqu/povAUaxWMBpy+ EoXagg7trzJ3JCLU3rhamo/bOLivLmZsWZlAAYkAxiy6JArUs3wHtL43A0xsn6z8 xuY3dBcCyGXawWkz9nR8+Z1ydy7xoVH/S1UPsiZiHd8z/exGwQn6S7YGBuYTmf6u CJscCFEo9WV+U7f51FI71zeNEnNBxRoZBg+ePVDurhmdUge4/2k3gp2GKJ0SkHWf sMgQ2UNJ+fYfTDSDboIeMlwiuhuXaS3iYZkAu2SlBNdhczjSCj7IfhZHbPfaZt3C pr1uiBGF/txKWLC6YT6qNHv5oG+TtamJcs6DHPExXvCAx3vQN0C/yA9uVg75ZxCP Xo0HgJALAmzqzjJ4wsB+zWMLvU1dC3lfIf7uxEQvWamX0/wpQGH+DuTM5GLBq0Lg wUeXD4MxZT05EOyJMfFK/NR3lbaTTxmil9CQmu3VrkxNw+jEJboBdmfdigcLNC9B 6XG46PcYQsSWHn2cWWvA =cJ1L -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54356049.9040009>