Date: Thu, 18 Oct 2012 09:39:34 -0400 From: John Baldwin <jhb@freebsd.org> To: freebsd-hackers@freebsd.org Cc: Konstantin Belousov <kostikbel@gmail.com>, Tristan Verniquet <tris_vern@hotmail.com> Subject: Re: syncing large mmaped files Message-ID: <201210180939.34861.jhb@freebsd.org> In-Reply-To: <20121018083537.GQ35915@deviant.kiev.zoral.com.ua> References: <SNT124-W20F26CF7B468F7F09B9B4983760@phx.gbl> <20121018083537.GQ35915@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday, October 18, 2012 4:35:37 am Konstantin Belousov wrote: > On Thu, Oct 18, 2012 at 10:08:22AM +1000, Tristan Verniquet wrote: > > > > I want to work with large (1-10G) files in memory but eventually sync > > them back out to disk. The problem is that the sync process appears to > > lock the file in kernel for the duration of the sync, which can run > > into minutes. This prevents other processes from reading from the file > > (unless they already have it mapped) for this whole time. Is there > > any way to prevent this? I think I read in a post somewhere about > > openbsd implementing partial-writes when it hits a file with lots of > > dirty pages in order to prevent this. Is there anything available for > > FreeBSD or is there another way around it? > > > No, currently the vnode lock is held exclusive for the whole duration > of the msync(2) syscall or its analog from the syncer. > > Making a change to periodically drop the vnode lock in > vm_object_page_clean() might be possible, but requires the benchmarking > to make sure that we do not pessimize the common case. Also, this opens > a possibility for the vnode reclamation meantime. You can simulate this in userland by breaking up your msync() into multiple msync() calls where each call just syncs a portion of the file. > Anyway, note that you cannot 'work with large files in memory', even if > you have enough RAM and no pressure to hold all the file pages resident. > The syncer will do a writeback periodically regardless of the application > calling msync(2) or not, with the interval of approximately 30 seconds. You can mmap with MAP_NOSYNC to prevent the syncer from writing the file out every 30 seconds. -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201210180939.34861.jhb>