From owner-freebsd-hackers@freebsd.org Thu Jul 16 15:42:31 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48CA39A38DC for ; Thu, 16 Jul 2015 15:42:31 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2740E1EB9 for ; Thu, 16 Jul 2015 15:42:31 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 263979A38DB; Thu, 16 Jul 2015 15:42:31 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 25C279A38DA for ; Thu, 16 Jul 2015 15:42:31 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mail-la0-x233.google.com (mail-la0-x233.google.com [IPv6:2a00:1450:4010:c03::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B28891EB8 for ; Thu, 16 Jul 2015 15:42:30 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by lagw2 with SMTP id w2so45893832lag.3 for ; Thu, 16 Jul 2015 08:42:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:message-id:date:to:mime-version; bh=PS4rGTqKTbv8Q0sYDu1Mf34GMzJtitHJE/eC9Y/bFdU=; b=Y6sJxz3vQZZA9tKR6dqsvXiKNjh1WEffeVg0ulty2MJp92JaA6B8bW3j+2T6E2A5cq jjUKTGsvfsapr54z296dJcYqOFHd1tK4I2CHVTb2a5v15LCRmbadwLZbl/Dk2yF8JHaS hlDMnHcMTkch4umBv2SjtIbSld3YGf42p8ApYuKriXNYoR0CJMCjYK6+cN1vb+GNgCz+ gS22K7d0/2wTLx35P5zVnSOC71c4hv115XQ46WJ0E3vK7+8ZO06ZwvVbUiSb4W6+ok6j VgPLPHrCzdFtjphIYEmQ4buI+/z4Sl1vhrDDZd79FODzEmojKrxjb+KZFcHE4tcSfWJu xVLQ== X-Received: by 10.112.27.138 with SMTP id t10mr9963934lbg.42.1437061348692; Thu, 16 Jul 2015 08:42:28 -0700 (PDT) Received: from 95.108.174.185-red.dhcp.yndx.net (95.108.174.185-red.dhcp.yndx.net. [95.108.174.185]) by smtp.gmail.com with ESMTPSA id si3sm2131847lbb.32.2015.07.16.08.42.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 16 Jul 2015 08:42:27 -0700 (PDT) From: Dmitry Sivachenko Content-Type: multipart/mixed; boundary="Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA" Subject: Strange memory management with mmap() Message-Id: Date: Thu, 16 Jul 2015 18:42:26 +0300 To: hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2015 15:42:31 -0000 --Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hello! I am using FreeBSD-10-stable and writing a program that uses large data = file via mmap() in read only mode. To be specific, I have 256GB RAM machine and typical size of data file = is ~160GB (more than 1/2 of RAM and less that the whole RAM). There is no other programs running during the test. Consider the following use case: I have two files on disk. I mmap() the = first one and prefetch data to RAM (touch every page of the file). After that I expect all data to be cached in RAM and subsequent access = will be fast. Next I do munmap() on the first file, mmap() the second one and do the = same test: prefetch data and expect it to be cached in RAM (and some of = the pages belonging to the first file to be purged out, because = size_of(file1)+size_of(file2) > size_of(RAM). Please find my test program attached. I run the program with 2 files provided via command line (both about = 160GB). What I observe in real is: -- before I run the program all RAM is in FREE state as reported by = top(1). -- after first prefetch() of the first file, all it's data goes to = "Cache" state, RES column of the process remains the same (small) -- second prefetch() works fast as expected, memory goes from Cache to = Active state, RES column of the process grows up to match file size = (SIZE=3D=3DRES now) -- now first prefetch() for second file starts: the remaining Free = memory goes to Cache state, Active size still equals to first file size. -- second prefetch() for second file works as slow as first one, like if = nothing was cached in memory during the first prefetch() run, RES column = does not change. Here is the output: % /tmp/a.out file1.dat file2.dat file1.dat... First prefault time: 1235.747351 seconds Second prefault time: 74.893323 seconds Ok. file2.dat... First prefault time: 1316.405527 seconds Second prefault time: 1311.491842 seconds Ok. I treat this like the bug somewhere in virtual memory management. Am I = right or my expectations about how that test program should work are = false? Thanks in advance. --Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA Content-Disposition: attachment; filename=mmap_test.c Content-Type: application/octet-stream; name="mmap_test.c" Content-Transfer-Encoding: 7bit #include #include #include #include #include #include #include #include static void prefault(const char *buf, size_t s) { volatile const char *p; size_t page_size; if (s == 0) return; page_size = sysconf(_SC_PAGESIZE); for (p = buf; p < buf + s; p += page_size) *p; *(volatile const char *)(buf + s - 1); } int main(int argc, char* argv[]) { if (argc < 2) { fprintf(stderr, "Usage: %s ...\n", argv[0]); exit(0); } int i, fd; struct stat st; void *p; struct timeval tv1, tv2; for (i=1; i < argc; i++) { printf("%s... ", argv[i]); if ((fd = open(argv[i], O_RDONLY)) < 0) err(1, "open"); if (fstat(fd, &st) != 0) err(1, "fstat"); if (st.st_size > 0) { if ((p = mmap(NULL, st.st_size, PROT_READ, MAP_NOCORE, fd, 0)) == MAP_FAILED) err(1, "mmap"); gettimeofday(&tv1, NULL); prefault(p, st.st_size); gettimeofday(&tv2, NULL); printf("First prefault time: %f seconds\n", (double) (tv2.tv_usec - tv1.tv_usec) / 1000000 + (double) (tv2.tv_sec - tv1.tv_sec)); gettimeofday(&tv1, NULL); prefault(p, st.st_size); gettimeofday(&tv2, NULL); printf("Second prefault time: %f seconds\n", (double) (tv2.tv_usec - tv1.tv_usec) / 1000000 + (double) (tv2.tv_sec - tv1.tv_sec)); if (munmap(p, st.st_size) != 0) err(1, "munmap"); } close(fd); printf("Ok.\n"); } return 0; } --Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA--