From owner-freebsd-hackers@freebsd.org Wed Jul 22 12:47:17 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C23A69A6264 for ; Wed, 22 Jul 2015 12:47:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A03101AF5 for ; Wed, 22 Jul 2015 12:47:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 9D87E9A6263; Wed, 22 Jul 2015 12:47:17 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 836AA9A6262 for ; Wed, 22 Jul 2015 12:47:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mail-la0-x22f.google.com (mail-la0-x22f.google.com [IPv6:2a00:1450:4010:c03::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 097441AF3 for ; Wed, 22 Jul 2015 12:47:17 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by laah7 with SMTP id h7so5870391laa.0 for ; Wed, 22 Jul 2015 05:47:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=UIj9LWq2By1/11YZ9O2yyqAoRqhmZ9AC4LE++iYJNd4=; b=wKg3enlMdKZHDNfN15PX1X4Zn+/Foy8upLcl59JsD8VZJgKj+ZOYs2fRF4HTLWytws KIG2v2/679hrJH0U3DkyPwqsmTaaQIfxuCcRgO9uo+EghgEBAtehx8cnzNFF/mtSxvcl yYg580pdPuJx25PzO5+YQU2UivdiAOpFoq/1zb9+pYQmzp4oCkIcy5aUwBP9B57W6MSd 4U4wI47IKqwauQ9dIOfo4Z3GlV8mR8EcwBQmQMc9jCf4v5VtLOLqM7cCRsZifxuJRfPn iqhPBqG+Puq+Z0u4QuQtSX9wR/Re824+JXzyEljgPjuPeWUBpVGp6ULS1R+jhlsRs11J 2l/A== X-Received: by 10.152.170.165 with SMTP id an5mr2222737lac.100.1437569234612; Wed, 22 Jul 2015 05:47:14 -0700 (PDT) Received: from ?IPv6:2a02:6b8::408:6ddc:be3:ab63:b221? ([2a02:6b8:0:408:6ddc:be3:ab63:b221]) by smtp.gmail.com with ESMTPSA id sg8sm278668lbb.33.2015.07.22.05.47.13 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 22 Jul 2015 05:47:13 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: Strange memory management with mmap() From: Dmitry Sivachenko In-Reply-To: <3434ED75-7994-4E9E-9B06-FACCD7DC90FF@gmail.com> Date: Wed, 22 Jul 2015 15:47:12 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <15DE3B94-3C09-4855-A274-D5655B049403@gmail.com> References: <3434ED75-7994-4E9E-9B06-FACCD7DC90FF@gmail.com> To: hackers@freebsd.org X-Mailer: Apple Mail (2.2102) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2015 12:47:17 -0000 > On 16 =D0=B8=D1=8E=D0=BB=D1=8F 2015 =D0=B3., at 21:19, Dmitry = Sivachenko wrote: >=20 >>=20 >> On 16 =D0=B8=D1=8E=D0=BB=D1=8F 2015 =D0=B3., at 18:42, Dmitry = Sivachenko wrote: >>=20 >> Hello! >>=20 >> I am using FreeBSD-10-stable and writing a program that uses large = data file via mmap() in read only mode. >> To be specific, I have 256GB RAM machine and typical size of data = file is ~160GB (more than 1/2 of RAM and less that the whole RAM). >> There is no other programs running during the test. >>=20 >> Consider the following use case: I have two files on disk. I mmap() = the first one and prefetch data to RAM (touch every page of the file). >> After that I expect all data to be cached in RAM and subsequent = access will be fast. >>=20 >> Next I do munmap() on the first file, mmap() the second one and do = the same test: prefetch data and expect it to be cached in RAM (and some = of the pages belonging to the first file to be purged out, because = size_of(file1)+size_of(file2) > size_of(RAM). >>=20 >> Please find my test program attached. >>=20 >> I run the program with 2 files provided via command line (both about = 160GB). >> What I observe in real is: >> -- before I run the program all RAM is in FREE state as reported by = top(1). >> -- after first prefetch() of the first file, all it's data goes to = "Cache" state, RES column of the process remains the same (small) >> -- second prefetch() works fast as expected, memory goes from Cache = to Active state, RES column of the process grows up to match file size = (SIZE=3D=3DRES now) >> -- now first prefetch() for second file starts: the remaining Free = memory goes to Cache state, Active size still equals to first file size. >> -- second prefetch() for second file works as slow as first one, like = if nothing was cached in memory during the first prefetch() run, RES = column does not change. >>=20 >>=20 >> Here is the output: >> % /tmp/a.out file1.dat file2.dat >> file1.dat... First prefault time: 1235.747351 seconds >> Second prefault time: 74.893323 seconds >> Ok. >> file2.dat... First prefault time: 1316.405527 seconds >> Second prefault time: 1311.491842 seconds >> Ok. >>=20 >=20 >=20 I tried the same test program on Linux machine with similar hardware. = It behaves like expected (second prefetch works very fast): file1.dat... First prefault time: 2664.621088 seconds Second prefault time: 1.969283 seconds Ok. file2.dat... First prefault time: 2917.009003 seconds Second prefault time: 34.128762 seconds Ok.