From owner-freebsd-hackers@freebsd.org Thu Jul 16 18:19:18 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 020B69A3B68 for ; Thu, 16 Jul 2015 18:19:18 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D514C1170 for ; Thu, 16 Jul 2015 18:19:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id D23799A3B67; Thu, 16 Jul 2015 18:19:17 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7E429A3B66 for ; Thu, 16 Jul 2015 18:19:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mail-la0-x231.google.com (mail-la0-x231.google.com [IPv6:2a00:1450:4010:c03::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F90A116F for ; Thu, 16 Jul 2015 18:19:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by laem6 with SMTP id m6so48405921lae.0 for ; Thu, 16 Jul 2015 11:19:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=YGesokFBKxDmqEMCGoX+OyJyeyZiqj3iSmphAsJjD6E=; b=fkyMHdK6oSVFW11l013d8kHfBNUvPD5RzhP05xeTvNfRUFzZFj9yKoVbRCZDA1nuJg vaqhSZKOwWJvcWtfKm1n6tYkxQqGO2wmAXnqmcRfaDYzf2V7v9ddebvH37t6ZIhV7v8u gU+OioLGS8KLu2oPvb2pxp3TF2xgqFvedbxv0UNBZj05ZR6uk5W7N5kOFjUvBJ7fK42c +D5Phqxl/9Vb3ziyiuUN4tewItXr6IxUYVXnolkgjuaHDVSSddxu/a73eoD2RHKAaseV 6tUe2hdUsj5XeuVY52cN0jXBEMkJ8LX/086PevTVyNkti65u5gJEUjFsM9ZtHrLuTi7Q 4xBg== X-Received: by 10.112.124.164 with SMTP id mj4mr10476884lbb.3.1437070755357; Thu, 16 Jul 2015 11:19:15 -0700 (PDT) Received: from [10.0.1.6] (broadband-5-228-251-172.nationalcablenetworks.ru. [5.228.251.172]) by smtp.gmail.com with ESMTPSA id ky9sm304694lab.49.2015.07.16.11.19.14 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 16 Jul 2015 11:19:14 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Strange memory management with mmap() From: Dmitry Sivachenko In-Reply-To: Date: Thu, 16 Jul 2015 21:19:13 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <3434ED75-7994-4E9E-9B06-FACCD7DC90FF@gmail.com> References: To: hackers@freebsd.org X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jul 2015 18:19:18 -0000 > On 16 =D0=B8=D1=8E=D0=BB=D1=8F 2015 =D0=B3., at 18:42, Dmitry = Sivachenko wrote: >=20 > Hello! >=20 > I am using FreeBSD-10-stable and writing a program that uses large = data file via mmap() in read only mode. > To be specific, I have 256GB RAM machine and typical size of data file = is ~160GB (more than 1/2 of RAM and less that the whole RAM). > There is no other programs running during the test. >=20 > Consider the following use case: I have two files on disk. I mmap() = the first one and prefetch data to RAM (touch every page of the file). > After that I expect all data to be cached in RAM and subsequent access = will be fast. >=20 > Next I do munmap() on the first file, mmap() the second one and do the = same test: prefetch data and expect it to be cached in RAM (and some of = the pages belonging to the first file to be purged out, because = size_of(file1)+size_of(file2) > size_of(RAM). >=20 > Please find my test program attached. >=20 > I run the program with 2 files provided via command line (both about = 160GB). > What I observe in real is: > -- before I run the program all RAM is in FREE state as reported by = top(1). > -- after first prefetch() of the first file, all it's data goes to = "Cache" state, RES column of the process remains the same (small) > -- second prefetch() works fast as expected, memory goes from Cache to = Active state, RES column of the process grows up to match file size = (SIZE=3D=3DRES now) > -- now first prefetch() for second file starts: the remaining Free = memory goes to Cache state, Active size still equals to first file size. > -- second prefetch() for second file works as slow as first one, like = if nothing was cached in memory during the first prefetch() run, RES = column does not change. >=20 >=20 > Here is the output: > % /tmp/a.out file1.dat file2.dat > file1.dat... First prefault time: 1235.747351 seconds > Second prefault time: 74.893323 seconds > Ok. > file2.dat... First prefault time: 1316.405527 seconds > Second prefault time: 1311.491842 seconds > Ok. >=20 And one more strange observation I made:=20 if I add=20 if (madvise(p, st.st_size, MADV_RANDOM) !=3D 0) err(1, "madvise"); if (madvise(p, st.st_size, MADV_WILLNEED) !=3D 0) err(1, "madvise"); after mmap() call in my test program, these times change as follows: % /tmp/a.out file1.dat file2.dat file1.dat... First prefault time: 2706.270454 seconds Second prefault time: 3.812012 seconds Ok. file2.dat... First prefault time: 4089.096647 seconds Second prefault time: 1132.210888 seconds Ok.