From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 5 00:38:16 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E42D146 for ; Fri, 5 Sep 2014 00:38:16 +0000 (UTC) Received: from out144-ams.mf.surf.net (out144-ams.mf.surf.net [145.0.1.144]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B07971C88 for ; Fri, 5 Sep 2014 00:38:15 +0000 (UTC) Received: from out43-ams.mf.surf.net (out43-ams.mf.surf.net [145.0.1.43]) by fbout1-ams.mf.surf.net (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id s850US9o018047 for ; Fri, 5 Sep 2014 02:30:28 +0200 Received: from smtps.utwente.nl (smtp-o1.utsp.utwente.nl [130.89.2.9]) by outgoing2-ams.mf.surf.net (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id s850UJIN029187 for ; Fri, 5 Sep 2014 02:30:19 +0200 Received: from [130.89.165.91] (nox.student.utwente.nl [130.89.165.91]) by smtps.utwente.nl (8.13.8) with ESMTP id s850UJWb028900 for ; Fri, 5 Sep 2014 02:30:19 +0200 Message-ID: <540903FF.6010602@degoeje.nl> Date: Fri, 05 Sep 2014 02:29:51 +0200 From: Pieter de Goeje User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: hackers@freebsd.org Subject: mmap MAP_NOSYNC regression in 10.x Content-Type: multipart/mixed; boundary="------------000800040209070700030907" X-Bayes-Prob: 0.0001 (Score 0, tokens from: utwente-out:default, base:default, @@RPTN) X-Spam-Score: -0.50 () [Tag at 5.00] 2450(0),CC(NL:-0.5) X-CanIt-Geo: ip=130.89.2.9; country=NL; region=Provincie Overijssel; city=Enschede; latitude=52.2195; longitude=6.8912; http://maps.google.com/maps?q=52.2195,6.8912&z=6 X-CanItPRO-Stream: utwente-out:default (inherits from utwente:default, base:default) X-Canit-Stats-ID: 0vMLcuj0C - b95822e8545d - 20140905 (trained as not-spam) X-Scanned-By: CanIt (www . roaringpenguin . com) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Sep 2014 00:38:16 -0000 This is a multi-part message in MIME format. --------------000800040209070700030907 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit After upgrading my month old 10-stable installation today (to r271093) , I've noticed a that the kernel no longer honors the MAP_NOSYNC flag. Attached is a demonstration program that highlights the issue (also available here: http://pastebin.com/y0kvdn0r ). The program creates and mmap()s a 200MiB file and repeatedly writes zeros to it. The expected behavior is that under normal circumstances (no memory pressure), the dirtied pages are not flushed to disk. Observed is however that every ~30 seconds the syncer kicks in and basically halts the program while it does its job. The program prints a line everytime the throughput drops below 500MBps, well below memory bandwidth. mmap() is called like this: void *p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_NOSYNC | MAP_ALIGNED_SUPER, fd, 0); Sample output: write... zeroing: 209.6 MB ...write: 5.839s mmap... ...mmap: 0.000s 20.1s: memset #259: 34.7MBps - stalled 55.7s: memset #810: 34.7MBps - stalled 91.3s: memset #1359: 34.6MBps - stalled 100.0s: memset #1522: 3938.5MBps overall bandwidth: 3190.6MBps munmap... ...munmap: 5.796s done (this is a rather old system) If necessary I'm willing to find out the exact commit that caused the problem. - Pieter --------------000800040209070700030907 Content-Type: text/plain; charset=windows-1252; name="mmap_test.c" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="mmap_test.c" #include #include #include #include #include #include #include #include #define CLEAREOL "\e[0K" #define CLOCK_START(a) double _clk_##a = now(); printf("%s... \n", #a) #define CLOCK_STOP(a) printf(" ...%s: %.3fs\n", #a, now() - _clk_##a) static double now() { struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); return ts.tv_sec + ts.tv_nsec * 1E-9; } int main() { setvbuf(stdout, NULL, _IONBF, 0); int fd = open("backing-file", O_RDWR | O_CREAT | O_TRUNC, 0666); if(fd == -1) err(1, "open"); size_t len = 200 * 1024 * 1024; size_t blen = 128 * 1024; CLOCK_START(write); char *buf = calloc(1, blen); for(size_t i = 0; i < len; i += blen) { printf("\rzeroing: %.1f MB" CLEAREOL, i * 1E-6); if(write(fd, buf, blen) != blen) err(1, "write\n"); } free(buf); printf("\n"); CLOCK_STOP(write); CLOCK_START(mmap); void *p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_NOSYNC | MAP_ALIGNED_SUPER, fd, 0); if(p == MAP_FAILED) err(1, "mmap"); CLOCK_STOP(mmap); if(close(fd) == -1) err(1, "close"); double start, stop, last; stop = start = now(); int n = 0; do { memset(p, 0, len); last = stop; stop = now(); double bw = 1E-6 * len / (stop - last); printf("\r%.1fs: memset #%d: %.1fMBps" CLEAREOL, stop - start, ++n, bw); if(bw < 500.0) printf(" - stalled\n"); } while(stop - start < 100.0); printf("\noverall bandwidth: %.1fMBps\n", n * (double)len / (stop - start) * 1E-6); CLOCK_START(munmap); if(munmap(p, len) == -1) err(1, "munmap"); CLOCK_STOP(munmap); printf("done\n"); return 0; } --------------000800040209070700030907--