From owner-freebsd-hackers@freebsd.org Wed Apr 27 20:48:41 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28CF2B1EE5E for ; Wed, 27 Apr 2016 20:48:41 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6B16139C for ; Wed, 27 Apr 2016 20:48:40 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: by mail-wm0-x22e.google.com with SMTP id e201so54176788wme.0 for ; Wed, 27 Apr 2016 13:48:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=date:from:to:subject:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=a7AMFuVvKSyWGYsN2iGJ3Kn7+gpfkJ7Qq2143GAgdKY=; b=J9XAfYu+jLeMRuG0G8PL3uDhQgtiElC0193LCbmE9I5Uebjp/8zMVx8Bq9FQ/zdtrd bNNTfnIdnJMSfhCCUzv59mSxJuLepEZLKBMx7qw1ey0fWXTPrpwUdWW/TcHgjYhoUxek WF1xaW/1jtlVMA6yUOW+QgKEAiyyPjyjoRcbSo8XcPxdMskMhw60Ol7hELGxt+dqh/Wu EtZOoGCWCq7nlas7wAK9F6Xzh7ECU0jON/c3HWysS7lrBH5xPAeyNqv6g6+VCJTiqTK/ Dc62FVRo1utAi6Nf94PdpXmoqthVU6oVjDl+AryXA/VcUDVZGSTyAm5uUZWRWIVApySw u69Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a7AMFuVvKSyWGYsN2iGJ3Kn7+gpfkJ7Qq2143GAgdKY=; b=Uon73DQSHBLc7bI/6vIbYhPpNB5+sx2gBJMWvvUq92rbo9ZsQBdX5FMgzuofMBePzm btfDBIzTFQHIKv0vSw6BnAIaXszmQ4tvIdzB4cSKGfRKH6aVI9prY45z7wam+QOUkzaR 7X8TYQwTQEWzwf+Jm2dPkIHDvKNm6jwZWaRcECP4QR1HqXv+R0W/UNVw6OMw8B0UNyDv 6UGYNWw9+af63YCK67v0om2sgjTjnTA4buPtXpS2KG95U8J2zcthPRjeVBqNuzPEaJuN Gl+rOJdtxyjCJwjTSFbxh8gwrj4oaOUh2AZTR8cmUstTfJJwLIl0AHcE49YHvp33cC/8 Lf1Q== X-Gm-Message-State: AOPr4FWlvZAO/2xGbTln2YtZtn9IWg62+lKMJDHbJ5PDNzuoE89RB8R/74DExVo1QJUEPQ== X-Received: by 10.194.143.51 with SMTP id sb19mr11530640wjb.98.1461790119407; Wed, 27 Apr 2016 13:48:39 -0700 (PDT) Received: from gumby.homeunix.com ([2.222.64.48]) by smtp.gmail.com with ESMTPSA id b15sm31105529wmd.1.2016.04.27.13.48.37 for (version=TLSv1/SSLv3 cipher=OTHER); Wed, 27 Apr 2016 13:48:38 -0700 (PDT) Date: Wed, 27 Apr 2016 21:48:36 +0100 From: RW To: freebsd-hackers@freebsd.org Subject: Re: Strange memory management with mmap() Message-ID: <20160427214836.05631eec@gumby.homeunix.com> In-Reply-To: References: X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd10.2) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2016 20:48:41 -0000 On Thu, 16 Jul 2015 18:42:26 +0300 Dmitry Sivachenko wrote: > I run the program with 2 files provided via command line (both about > 160GB). What I observe in real is: > -- before I run the program all RAM is in FREE state as reported by > top(1). -- after first prefetch() of the first file, all it's data > goes to "Cache" state, This is what surprises me most, I'd expect it to go to active. I thought that the point of the cache queue is that its pages can be reassigned because they are clean and not mapped anywhere. >RES column of the process remains the same > (small) -- second prefetch() works fast as expected, memory goes from > Cache to Active state, RES column of the process grows up to match > file size (SIZE==RES now) -- now first prefetch() for second file > starts: the remaining Free memory goes to Cache state, Active size > still equals to first file size. -- second prefetch() for second file > works as slow as first one, like if nothing was cached in memory > during the first prefetch() run, RES column does not change. > > Here is the output: > % /tmp/a.out file1.dat file2.dat > file1.dat... First prefault time: 1235.747351 seconds > Second prefault time: 74.893323 seconds > Ok. > file2.dat... First prefault time: 1316.405527 seconds > Second prefault time: 1311.491842 seconds > Ok. > > I treat this like the bug somewhere in virtual memory management. The timings do make sense. The management of the active queue isn't LRU, it takes account of how many times a page gets accessed. Whatever is going on with the cache queue doesn't seem to change that general strategy. The double prefail on the first file means that all its pages are in memory and have recently been accessed twice. When the system runs out of free memory the VM system retains the file1 pages at the expense of the pages from file2 which have only been accessed once. At the end of the first pass only the end of file2 is cached, which isn't any use on the second pass because the prefault works upwards. This is an uncommon pattern of access where simple LRU expiry of the cache will outperform anything more sophisticated. I don't think it matters that Linux does better on this particular benchmark.