From owner-freebsd-hackers@FreeBSD.ORG Tue Apr 3 19:02:56 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ED3741065670 for ; Tue, 3 Apr 2012 19:02:56 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 72CA58FC08 for ; Tue, 3 Apr 2012 19:02:56 +0000 (UTC) Received: by bkcjc3 with SMTP id jc3so50170bkc.13 for ; Tue, 03 Apr 2012 12:02:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:x-gm-message-state; bh=AlVasNTAZb+97HKzZR3v+G4SqJJjNHg7e6LexxrIzGg=; b=M48s+6KIBwwU1ZdgjrIIbsdVy6QNaIbICTyOdXy39qMkZkWSjgg4A3RWzyokaXvCGU Eqgw9F2FiiG4Pj3Wq1M7g1C16YfGDzmD1UrsksoDJjnx3K+431Y5BkvYvmlpuRMSi7aM 0EUzrQQiKQalP1LG9MXWvM+rEC9v9nS96Xlx9yB6ezvuKExnCWzcau6hW1Og4f6Qe8Fy aJ19yQ4D57yrrsQOWoamIKx63oYoWY/ZYdUu+4ANUkl5Q3hOIHFekROa8KlUzgn1XnoX 0HTf5ArK3oQs9hSRsNpfI/uS2F2s5sTmXKM5b6+IuTOeqNmgQgTzY2wWymtuRF72V6wQ PefA== Received: by 10.205.127.130 with SMTP id ha2mr6157479bkc.28.1333479775384; Tue, 03 Apr 2012 12:02:55 -0700 (PDT) Received: from [10.254.254.77] (ppp95-165-133-149.pppoe.spdop.ru. [95.165.133.149]) by mx.google.com with ESMTPS id jd17sm47960905bkb.4.2012.04.03.12.02.54 (version=SSLv3 cipher=OTHER); Tue, 03 Apr 2012 12:02:54 -0700 (PDT) Message-ID: <4F7B495D.3010402@zonov.org> Date: Tue, 03 Apr 2012 23:02:53 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: multipart/mixed; boundary="------------020308060409060606070001" X-Gm-Message-State: ALoCoQn7VKuqtMnwC6m2B+o/bekQK+DdFHwM36BHbbec9btnc6QDkpT/p7iqMjj/GmZQMt1kkJVL Subject: problems with mmap() and disk caching X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Apr 2012 19:02:57 -0000 This is a multi-part message in MIME format. --------------020308060409060606070001 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, I open the file, then call mmap() on the whole file and get pointer, then I work with this pointer. I expect that page should be only once touched to get it into the memory (disk cache?), but this doesn't work! I wrote the test (attached) and ran it for the 1G file generated from /dev/random, the result is the following: Prepare file: # swapoff -a # newfs /dev/ada0b # mount /dev/ada0b /mnt # dd if=/dev/random of=/mnt/random-1024 bs=1m count=1024 Purge cache: # umount /mnt # mount /dev/ada0b /mnt Run test: $ ./mmap /mnt/random-1024 30 mmap: 1 pass took: 7.431046 (none: 262112; res: 32; super: 0; other: 0) mmap: 2 pass took: 7.356670 (none: 261648; res: 496; super: 0; other: 0) mmap: 3 pass took: 7.307094 (none: 260521; res: 1623; super: 0; other: 0) mmap: 4 pass took: 7.350239 (none: 258904; res: 3240; super: 0; other: 0) mmap: 5 pass took: 7.392480 (none: 257286; res: 4858; super: 0; other: 0) mmap: 6 pass took: 7.292069 (none: 255584; res: 6560; super: 0; other: 0) mmap: 7 pass took: 7.048980 (none: 251142; res: 11002; super: 0; other: 0) mmap: 8 pass took: 6.899387 (none: 247584; res: 14560; super: 0; other: 0) mmap: 9 pass took: 7.190579 (none: 242992; res: 19152; super: 0; other: 0) mmap: 10 pass took: 6.915482 (none: 239308; res: 22836; super: 0; other: 0) mmap: 11 pass took: 6.565909 (none: 232835; res: 29309; super: 0; other: 0) mmap: 12 pass took: 6.423945 (none: 226160; res: 35984; super: 0; other: 0) mmap: 13 pass took: 6.315385 (none: 208555; res: 53589; super: 0; other: 0) mmap: 14 pass took: 6.760780 (none: 192805; res: 69339; super: 0; other: 0) mmap: 15 pass took: 5.721513 (none: 174497; res: 87647; super: 0; other: 0) mmap: 16 pass took: 5.004424 (none: 155938; res: 106206; super: 0; other: 0) mmap: 17 pass took: 4.224926 (none: 135639; res: 126505; super: 0; other: 0) mmap: 18 pass took: 3.749608 (none: 117952; res: 144192; super: 0; other: 0) mmap: 19 pass took: 3.398084 (none: 99066; res: 163078; super: 0; other: 0) mmap: 20 pass took: 3.029557 (none: 74994; res: 187150; super: 0; other: 0) mmap: 21 pass took: 2.379430 (none: 55231; res: 206913; super: 0; other: 0) mmap: 22 pass took: 2.046521 (none: 40786; res: 221358; super: 0; other: 0) mmap: 23 pass took: 1.152797 (none: 30311; res: 231833; super: 0; other: 0) mmap: 24 pass took: 0.972617 (none: 16196; res: 245948; super: 0; other: 0) mmap: 25 pass took: 0.577515 (none: 8286; res: 253858; super: 0; other: 0) mmap: 26 pass took: 0.380738 (none: 3712; res: 258432; super: 0; other: 0) mmap: 27 pass took: 0.253583 (none: 1193; res: 260951; super: 0; other: 0) mmap: 28 pass took: 0.157508 (none: 0; res: 262144; super: 0; other: 0) mmap: 29 pass took: 0.156169 (none: 0; res: 262144; super: 0; other: 0) mmap: 30 pass took: 0.156550 (none: 0; res: 262144; super: 0; other: 0) If I ran this: $ cat /mnt/random-1024 > /dev/null before test, when result is the following: $ ./mmap /mnt/random-1024 5 mmap: 1 pass took: 0.337657 (none: 0; res: 262144; super: 0; other: 0) mmap: 2 pass took: 0.186137 (none: 0; res: 262144; super: 0; other: 0) mmap: 3 pass took: 0.186132 (none: 0; res: 262144; super: 0; other: 0) mmap: 4 pass took: 0.186535 (none: 0; res: 262144; super: 0; other: 0) mmap: 5 pass took: 0.190353 (none: 0; res: 262144; super: 0; other: 0) This is what I expect. But why this doesn't work without reading file manually? I've also never seen super pages, how to make them work? I've been playing with madvise and posix_fadvise but no luck. BTW, posix_fadvise(POSIX_FADV_WILLNEED) does nothing as the commentary says, shouldn't this be documented in the manual page? All tests were run under 9.0-STABLE (r233744). -- Andrey Zonov --------------020308060409060606070001 Content-Type: text/plain; charset=windows-1251; name="mmap.c" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="mmap.c" /*_ * Andrey Zonov (c) 2011 */ #include #include #include #include #include #include #include #include #include int main(int argc, char **argv) { int i; int fd; int num; int block; int pagesize; size_t n; size_t size; size_t none, incore, super, other; char *p; char *tmp; char *vec; char *vecp; struct stat sb; struct timeval tp, tp1, tp2; if (argc < 2 || argc > 4) errx(1, "usage: mmap [num] [block]"); fd = open(argv[1], O_RDONLY); if (fd == -1) err(1, "open()"); num = 1; if (argc >= 3) num = atoi(argv[2]); pagesize = getpagesize(); block = pagesize; if (argc == 4) block = atoi(argv[3]); if (fstat(fd, &sb) == -1) err(1, "fstat()"); size = sb.st_size; #if 0 if (posix_fadvise(fd, (off_t)0, (off_t)0, POSIX_FADV_WILLNEED) == -1) err(1, "posix_fadvise()"); #endif p = mmap(NULL, sb.st_size, PROT_READ, /*MAP_PREFAULT_READ |*/ MAP_PRIVATE, fd, (off_t)0); if (p == MAP_FAILED) err(1, "mmap()"); #if 0 if (madvise(p, (size_t)size, MADV_WILLNEED) == -1) err(1, "madvise()"); #endif tmp = calloc(1, block); if (tmp == NULL) err(1, "calloc()"); vec = calloc(1, size / pagesize); if (vec == NULL) err(1, "calloc()"); for (i = 0; i < num; i++) { gettimeofday(&tp1, NULL); for (n = 0; n < size / block; n++) memcpy(tmp, p + (n * block), block); gettimeofday(&tp2, NULL); timersub(&tp2, &tp1, &tp); if (mincore(p, size, vec) == -1) err(1, "mincore()"); none = incore = super = other = 0; for (vecp = vec; (size_t)(vecp - vec) < size / pagesize; vecp++) { if (*vecp == 0) none++; else if (*vecp & MINCORE_INCORE) incore++; else if (*vecp & MINCORE_SUPER) super++; else other++; } warnx("%2d pass took: %3ld.%06ld (none: %6ld; res: %6ld; super: %6ld; other: %6ld)", i + 1, tp.tv_sec, tp.tv_usec, none, incore, super, other); } free(vec); free(tmp); if (munmap(p, sb.st_size) == -1) err(1, "munmap()"); close(fd); exit(0); } --------------020308060409060606070001--