From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 4 09:36:36 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2D9C01065678 for ; Wed, 4 Apr 2012 09:36:36 +0000 (UTC) (envelope-from andrey@zonov.org) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id A15348FC1D for ; Wed, 4 Apr 2012 09:36:35 +0000 (UTC) Received: by bkcjc3 with SMTP id jc3so87917bkc.13 for ; Wed, 04 Apr 2012 02:36:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding :x-gm-message-state; bh=xyefyInvH+zOqwd0cIEs9AvRPyOh45IEw5eg6ueLN18=; b=TudNV/y/JQoTRO0Uzuqzv1H92zcYlskHwZS0mICy1OertRChSPVzFEmXwMJWpsNqpT c3j8U3H0KBntk1yyme0BXnKhNokEQTbNVTtCtSO7kH5dgFjAkWeouxCrREpZXUKY25eT NiewupJVKzCv0QFu+e/8wk5VtRiovW8Rx8+yruCZJ8nfOfAFTa0NEBtyKBh7zntLRZv6 3n3hGL/pzpltos2fn3GPb/5x8cf0LyLciQPVV3q3B5+O9IWr9/HD0x/+lhlF5O81N9v4 l2n750bT06567t0ztcC20xu2r9KRsOyFBvJ9ou1AifAqNRCizjsEVaA230MyNnUb6u0I EE7g== Received: by 10.205.132.73 with SMTP id ht9mr7266688bkc.46.1333532194398; Wed, 04 Apr 2012 02:36:34 -0700 (PDT) Received: from [10.254.254.77] (ppp95-165-133-149.pppoe.spdop.ru. [95.165.133.149]) by mx.google.com with ESMTPS id u16sm332189bkf.10.2012.04.04.02.36.33 (version=SSLv3 cipher=OTHER); Wed, 04 Apr 2012 02:36:34 -0700 (PDT) Message-ID: <4F7C1620.6040703@zonov.org> Date: Wed, 04 Apr 2012 13:36:32 +0400 From: Andrey Zonov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Konstantin Belousov References: <4F7B495D.3010402@zonov.org> <20120404071746.GJ2358@deviant.kiev.zoral.com.ua> In-Reply-To: <20120404071746.GJ2358@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Gm-Message-State: ALoCoQnBpWyu5V1CH6gP3FT3PI+AuvplEfq1FHXQM9NKAJrQaQ+2ZRPP81gkCgrn5EtfHN6ZhunN Cc: alc@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: problems with mmap() and disk caching X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Apr 2012 09:36:36 -0000 On 04.04.2012 11:17, Konstantin Belousov wrote: > > Calling madvise(MADV_RANDOM) fixes the issue, because the code to > deactivate/cache the pages is turned off. On the other hand, it also > turns of read-ahead for faulting, and the first loop becomes eternally > long. Now it takes 5 times longer. Anyway, thanks for explanation. > > Doing MADV_WILLNEED does not fix the problem indeed, since willneed > reactivates the pages of the object at the time of call. To use > MADV_WILLNEED, you would need to call it between faults/memcpy. > I played with it, but no luck so far. >> >> I've also never seen super pages, how to make them work? > They just work, at least for me. Look at the output of procstat -v > after enough loops finished to not cause disk activity. > The problem was in my test program. I fixed it, now I see super pages but I'm still not satisfied. There are several tests below: 1. With madvise(MADV_RANDOM) I see almost all super pages: $ ./mmap /mnt/random-1024 5 mmap: 1 pass took: 26.438535 (none: 0; res: 262144; super: 511; other: 0) mmap: 2 pass took: 0.187311 (none: 0; res: 262144; super: 511; other: 0) mmap: 3 pass took: 0.184953 (none: 0; res: 262144; super: 511; other: 0) mmap: 4 pass took: 0.186007 (none: 0; res: 262144; super: 511; other: 0) mmap: 5 pass took: 0.185790 (none: 0; res: 262144; super: 511; other: 0) Should it be 512? 2. Without madvise(MADV_RANDOM): $ ./mmap /mnt/random-1024 50 mmap: 1 pass took: 7.629745 (none: 262112; res: 32; super: 0; other: 0) mmap: 2 pass took: 7.301720 (none: 261202; res: 942; super: 0; other: 0) mmap: 3 pass took: 7.261416 (none: 260226; res: 1918; super: 1; other: 0) [skip] mmap: 49 pass took: 0.155368 (none: 0; res: 262144; super: 323; other: 0) mmap: 50 pass took: 0.155438 (none: 0; res: 262144; super: 323; other: 0) Only 323 pages. 3. If I just re-run test I don't see super pages with any size of "block". $ ./mmap /mnt/random-1024 5 $((1<<30)) mmap: 1 pass took: 1.013939 (none: 0; res: 262144; super: 0; other: 0) mmap: 2 pass took: 0.267082 (none: 0; res: 262144; super: 0; other: 0) mmap: 3 pass took: 0.270711 (none: 0; res: 262144; super: 0; other: 0) mmap: 4 pass took: 0.268940 (none: 0; res: 262144; super: 0; other: 0) mmap: 5 pass took: 0.269634 (none: 0; res: 262144; super: 0; other: 0) 4. If I activate madvise(MADV_WILLNEDD) in the copy loop and re-run test then I see super pages only if I use "block" greater than 2Mb. $ ./mmap /mnt/random-1024 1 $((1<<21)) mmap: 1 pass took: 0.299722 (none: 0; res: 262144; super: 0; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<22)) mmap: 1 pass took: 0.271828 (none: 0; res: 262144; super: 170; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<23)) mmap: 1 pass took: 0.333188 (none: 0; res: 262144; super: 258; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<24)) mmap: 1 pass took: 0.339250 (none: 0; res: 262144; super: 303; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<25)) mmap: 1 pass took: 0.418812 (none: 0; res: 262144; super: 324; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<26)) mmap: 1 pass took: 0.360892 (none: 0; res: 262144; super: 335; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<27)) mmap: 1 pass took: 0.401122 (none: 0; res: 262144; super: 342; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<28)) mmap: 1 pass took: 0.478764 (none: 0; res: 262144; super: 345; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<29)) mmap: 1 pass took: 0.607266 (none: 0; res: 262144; super: 346; other: 0) $ ./mmap /mnt/random-1024 1 $((1<<30)) mmap: 1 pass took: 0.901269 (none: 0; res: 262144; super: 347; other: 0) 5. If I activate madvise(MADV_WILLNEED) immediately after mmap() then I see some number of super pages (the number from test #2). $ ./mmap /mnt/random-1024 5 mmap: 1 pass took: 0.178666 (none: 0; res: 262144; super: 323; other: 0) mmap: 2 pass took: 0.158889 (none: 0; res: 262144; super: 323; other: 0) mmap: 3 pass took: 0.157229 (none: 0; res: 262144; super: 323; other: 0) mmap: 4 pass took: 0.156895 (none: 0; res: 262144; super: 323; other: 0) mmap: 5 pass took: 0.162938 (none: 0; res: 262144; super: 323; other: 0) 6. If I read file manually before test then I don't see super pages with any size of "block" and madvise(MADV_WILLNEED) doesn't help. $ ./mmap /mnt/random-1024 5 $((1<<30)) mmap: 1 pass took: 0.996767 (none: 0; res: 262144; super: 0; other: 0) mmap: 2 pass took: 0.311129 (none: 0; res: 262144; super: 0; other: 0) mmap: 3 pass took: 0.317430 (none: 0; res: 262144; super: 0; other: 0) mmap: 4 pass took: 0.314437 (none: 0; res: 262144; super: 0; other: 0) mmap: 5 pass took: 0.310757 (none: 0; res: 262144; super: 0; other: 0) -- Andrey Zonov