From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 19:43:28 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EC9F7CDE for ; Thu, 18 Oct 2012 19:43:28 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id BD6838FC08 for ; Thu, 18 Oct 2012 19:43:28 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 28EA3B986; Thu, 18 Oct 2012 15:43:28 -0400 (EDT) From: John Baldwin To: Konstantin Belousov Subject: Re: syncing large mmaped files Date: Thu, 18 Oct 2012 15:43:25 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p20; KDE/4.5.5; amd64; ; ) References: <201210180939.34861.jhb@freebsd.org> <20121018164218.GR35915@deviant.kiev.zoral.com.ua> In-Reply-To: <20121018164218.GR35915@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201210181543.25191.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 18 Oct 2012 15:43:28 -0400 (EDT) Cc: freebsd-hackers@freebsd.org, Tristan Verniquet X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 19:43:29 -0000 On Thursday, October 18, 2012 12:42:18 pm Konstantin Belousov wrote: > On Thu, Oct 18, 2012 at 09:39:34AM -0400, John Baldwin wrote: > > On Thursday, October 18, 2012 4:35:37 am Konstantin Belousov wrote: > > > On Thu, Oct 18, 2012 at 10:08:22AM +1000, Tristan Verniquet wrote: > > > > > > > > I want to work with large (1-10G) files in memory but eventually sync > > > > them back out to disk. The problem is that the sync process appears to > > > > lock the file in kernel for the duration of the sync, which can run > > > > into minutes. This prevents other processes from reading from the file > > > > (unless they already have it mapped) for this whole time. Is there > > > > any way to prevent this? I think I read in a post somewhere about > > > > openbsd implementing partial-writes when it hits a file with lots of > > > > dirty pages in order to prevent this. Is there anything available for > > > > FreeBSD or is there another way around it? > > > > > > > No, currently the vnode lock is held exclusive for the whole duration > > > of the msync(2) syscall or its analog from the syncer. > > > > > > Making a change to periodically drop the vnode lock in > > > vm_object_page_clean() might be possible, but requires the benchmarking > > > to make sure that we do not pessimize the common case. Also, this opens > > > a possibility for the vnode reclamation meantime. > > > > You can simulate this in userland by breaking up your msync() into multiple > > msync() calls where each call just syncs a portion of the file. > Be aware that this is much-much slower than msyncing the whole file, even > if file is very large. The reason is that pager initiates asynchronous > _immediate_ clustered write for such situations. Async writes (AKA > bdwrite()) are only specified for full range msyncing. Ugh. It would seem to me that msync(MS_ASYNC) should be doing delayed writes. > > > Anyway, note that you cannot 'work with large files in memory', even if > > > you have enough RAM and no pressure to hold all the file pages resident. > > > The syncer will do a writeback periodically regardless of the application > > > calling msync(2) or not, with the interval of approximately 30 seconds. > > > > You can mmap with MAP_NOSYNC to prevent the syncer from writing the file out > > every 30 seconds. > > This also prevents msync(2) from syncing the region. The flag is fine > for throw-away data, but not for the scenario that was described, I > think. Oof. I could see that in certain situations you might want to control this behavior from an application (similar to how I now make use of fadvise() at work). Having a way to disable syncer but having msync(MS_ASYNC) do something useful would be good. -- John Baldwin