From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 08:35:45 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D123940F for ; Thu, 18 Oct 2012 08:35:45 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 4AEC28FC08 for ; Thu, 18 Oct 2012 08:35:44 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q9I8Zn6c015975; Thu, 18 Oct 2012 11:35:49 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q9I8Zbbc004754; Thu, 18 Oct 2012 11:35:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q9I8Zbul004753; Thu, 18 Oct 2012 11:35:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 18 Oct 2012 11:35:37 +0300 From: Konstantin Belousov To: Tristan Verniquet Subject: Re: syncing large mmaped files Message-ID: <20121018083537.GQ35915@deviant.kiev.zoral.com.ua> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="XA7quakUSnawneuz" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 08:35:45 -0000 --XA7quakUSnawneuz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Oct 18, 2012 at 10:08:22AM +1000, Tristan Verniquet wrote: >=20 > I want to work with large (1-10G) files in memory but eventually sync > them back out to disk. The problem is that the sync process appears to > lock the file in kernel for the duration of the sync, which can run > into minutes. This prevents other processes from reading from the file > (unless they already have it mapped) for this whole time. Is there > any way to prevent this? I think I read in a post somewhere about > openbsd implementing partial-writes when it hits a file with lots of > dirty pages in order to prevent this. Is there anything available for > FreeBSD or is there another way around it? > No, currently the vnode lock is held exclusive for the whole duration of the msync(2) syscall or its analog from the syncer. Making a change to periodically drop the vnode lock in vm_object_page_clean() might be possible, but requires the benchmarking to make sure that we do not pessimize the common case. Also, this opens a possibility for the vnode reclamation meantime. Anyway, note that you cannot 'work with large files in memory', even if you have enough RAM and no pressure to hold all the file pages resident. The syncer will do a writeback periodically regardless of the application calling msync(2) or not, with the interval of approximately 30 seconds. --XA7quakUSnawneuz Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlB/v1kACgkQC3+MBN1Mb4joagCgj1oYiDMQjM9s9kK7HniP4JiL RVEAn1294Rq3lIUMnPdt2G2ue1z3Jppa =Z1TH -----END PGP SIGNATURE----- --XA7quakUSnawneuz--