From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 9 11:42:30 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2ACF239B for ; Wed, 9 Oct 2013 11:42:30 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mail-lb0-x22d.google.com (mail-lb0-x22d.google.com [IPv6:2a00:1450:4010:c04::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id ACA042FA9 for ; Wed, 9 Oct 2013 11:42:29 +0000 (UTC) Received: by mail-lb0-f173.google.com with SMTP id o14so649989lbi.18 for ; Wed, 09 Oct 2013 04:42:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:subject:message-id:date :to:mime-version; bh=DQQdE9vKTZGi3H3ldxEqVSpOJsc/H5x/Rnc+n3r6UWE=; b=qq1yZiKbgvHgQqJp8aPMVQMOoJrSHF/tMK7jYHDXd86/zco3BYZ1e2vC+/dkt8HlKi xDA9jYjJ+6gbWWfX5Y/Vovz4m5GO3gvnTHJ/3DmvbRR0Y/mdrUfAPylhwu7GU4UCY7hA pzvaOCSrIuMgIRs1vYJPu8Rcu2L/2lBQgKCIz5A0OOrbFpnM3+hRB8FJQw0TIaRpRnf6 uwBi25lSHHNKpgwTNX66B7/+j35Vp2bQjAF3DBdrYUlWyxJU+JldY7kyLCwejWRK5RnW EzzE6xXd3zKMaYRG7XyuQzBfj00yyK4iyt/TJJmpUJp8e4nYGvDmHl4vjNSL34dOuhJH swtg== X-Received: by 10.152.170.166 with SMTP id an6mr6037817lac.20.1381318947662; Wed, 09 Oct 2013 04:42:27 -0700 (PDT) Received: from dhcp174-208-red.yandex.net (dhcp174-208-red.yandex.net. [95.108.174.208]) by mx.google.com with ESMTPSA id i3sm34759886laf.4.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 09 Oct 2013 04:42:26 -0700 (PDT) From: Dmitry Sivachenko Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: mmap() question Message-Id: <95E0B821-BF9B-4EBF-A1E5-1DDCBB1C3D1B@gmail.com> Date: Wed, 9 Oct 2013 15:42:27 +0400 To: "hackers@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) X-Mailer: Apple Mail (2.1510) X-Mailman-Approved-At: Wed, 09 Oct 2013 12:07:19 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Oct 2013 11:42:30 -0000 Hello! I have a program which mmap()s a lot of large files (total size more = that RAM and I have no swap), but it needs only small parts of that = files at a time. My understanding is that when using mmap when I access some memory = region OS reads the relevant portion of that file from disk and caches = the result in memory. If there is no free memory, OS will purge = previously read part of mmap'ed file to free memory for the new chunk. But this is not the case. I use the following simple program which gets = list of files as command line arguments, mmap()s them all and then = selects random file and random 1K parts of that file and computes a XOR = of bytes from that region. After some time the program dies: pid 63251 (a.out), uid 1232, was killed: out of swap space It seems I incorrectly understand how mmap() works, can you please = clarify what's going wrong? I expect that program to run indefinitely, purging some regions out of = RAM and reading the relevant parts of files. Thanks! #include #include #include #include #include #include #include #include #include struct f_data { char *beg; off_t size; }; int main(int argc, char* argv[]) { if (argc < 2) { fprintf(stderr, "Usage: %s ...\n", argv[0]); exit(0); } int i, j, fd; struct stat st; struct f_data FILES[500]; int NUM_FILES; void *p; NUM_FILES =3D argc - 1; for (i=3D1; i < argc; i++) { printf("%s... ", argv[i]); if ((fd =3D open(argv[i], O_RDONLY)) < 0) errx(1, "open"); if (fstat(fd, &st) !=3D 0) errx(1, "fstat"); if ((p =3D mmap(NULL, st.st_size, PROT_READ, MAP_NOCORE, fd, 0)) = =3D=3D MAP_FAILED) errx(1, "mmap"); FILES[i-1].beg =3D (char*)p; FILES[i-1].size =3D st.st_size; if (msync(p, st.st_size, MS_INVALIDATE) !=3D 0) errx(1, "msync"); printf("Ok.\n"); } char chk =3D 0; while(1) { int rf =3D floor((double)random() / 2147483647 * NUM_FILES); off_t offs =3D floor((double)random() / 2147483647 * = (FILES[rf].size - 1024)); for (j=3D0; j<1024; j++) chk ^=3D *(FILES[rf].beg + offs + j); } return 0; }