From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 15:17:11 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E9EA0A54; Thu, 18 Oct 2012 15:17:11 +0000 (UTC) (envelope-from tris_vern@hotmail.com) Received: from snt0-omc3-s36.snt0.hotmail.com (snt0-omc3-s36.snt0.hotmail.com [65.55.90.175]) by mx1.freebsd.org (Postfix) with ESMTP id B638B8FC08; Thu, 18 Oct 2012 15:17:11 +0000 (UTC) Received: from SNT124-W29 ([65.55.90.135]) by snt0-omc3-s36.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 18 Oct 2012 08:16:04 -0700 Message-ID: X-Originating-IP: [165.228.7.150] From: Tristan Verniquet To: , freebsd hackers Subject: RE: syncing large mmaped files Date: Fri, 19 Oct 2012 01:16:04 +1000 Importance: Normal In-Reply-To: <201210180939.34861.jhb@freebsd.org> References: , <20121018083537.GQ35915@deviant.kiev.zoral.com.ua>, <201210180939.34861.jhb@freebsd.org> MIME-Version: 1.0 X-OriginalArrivalTime: 18 Oct 2012 15:16:04.0711 (UTC) FILETIME=[7DB60F70:01CDAD43] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: kostikbel@gmail.com X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 15:17:12 -0000 > From: jhb@freebsd.org > To: freebsd-hackers@freebsd.org > Subject: Re: syncing large mmaped files > Date: Thu=2C 18 Oct 2012 09:39:34 -0400 > CC: kostikbel@gmail.com=3B tris_vern@hotmail.com >=20 > On Thursday=2C October 18=2C 2012 4:35:37 am Konstantin Belousov wrote: > > On Thu=2C Oct 18=2C 2012 at 10:08:22AM +1000=2C Tristan Verniquet wrote= : > > >=20 > > > I want to work with large (1-10G) files in memory but eventually sync > > > them back out to disk. The problem is that the sync process appears t= o > > > lock the file in kernel for the duration of the sync=2C which can run > > > into minutes. This prevents other processes from reading from the fil= e > > > (unless they already have it mapped) for this whole time. Is there > > > any way to prevent this? I think I read in a post somewhere about > > > openbsd implementing partial-writes when it hits a file with lots of > > > dirty pages in order to prevent this. Is there anything available for > > > FreeBSD or is there another way around it? > > > > > No=2C currently the vnode lock is held exclusive for the whole duration > > of the msync(2) syscall or its analog from the syncer. > >=20 > > Making a change to periodically drop the vnode lock in > > vm_object_page_clean() might be possible=2C but requires the benchmarki= ng > > to make sure that we do not pessimize the common case. Also=2C this ope= ns > > a possibility for the vnode reclamation meantime. >=20 > You can simulate this in userland by breaking up your msync() into multip= le > msync() calls where each call just syncs a portion of the file. Thanks=2C I was doing this and I thought I was getting much worse performan= ce from the msync over the fsync=2C however I am trying it again now and th= e difference doesn't seem as large as I first imagined. It is still taking = about 4x as long for the case where all the pages are dirty but catches up = when the file is more sparsely written. I guess that is probably acceptable= . When all pages are dirty=2C iostat shows that the fsync will write 128KB/Tr= ansaction=2C whereas msync always does 16 KB/Transaction and a lower MB/s. = It will continue to do this if I only dirty every 2nd=2C 3rd or 4th page. W= hen I only dirty every 5th page the fsync seems to kick into another mode a= nd starts doing 16KB/Transaction and the time starts becoming comparable to= msync. Is there anyway to get that fsync 128K/Transaction performance increase whe= n all pages are dirty with msync?=20 > > Anyway=2C note that you cannot 'work with large files in memory'=2C eve= n if > > you have enough RAM and no pressure to hold all the file pages resident= . > > The syncer will do a writeback periodically regardless of the applicati= on > > calling msync(2) or not=2C with the interval of approximately 30 second= s. >=20 > You can mmap with MAP_NOSYNC to prevent the syncer from writing the file = out > every 30 seconds. Yes=2C I was mapping MAP_NOSYNC. =20 > --=20 > John Baldwin > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe=2C send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" =