From owner-freebsd-fs@FreeBSD.ORG Wed Jul 4 09:06:42 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E165E106566B for ; Wed, 4 Jul 2012 09:06:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 5FBF98FC0C for ; Wed, 4 Jul 2012 09:06:41 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q6496kYm086363; Wed, 4 Jul 2012 12:06:46 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q6496Xaf012855; Wed, 4 Jul 2012 12:06:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q6496XTu012854; Wed, 4 Jul 2012 12:06:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 4 Jul 2012 12:06:33 +0300 From: Konstantin Belousov To: Pavlo Message-ID: <20120704090633.GH2337@deviant.kiev.zoral.com.ua> References: <91943.1339669820.1305529125424791552@ffe15.ukr.net> <23856.1341389256.6316487571580649472@ffe17.ukr.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vX2ve0hYIA9A871f" Content-Disposition: inline In-Reply-To: <23856.1341389256.6316487571580649472@ffe17.ukr.net> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org Subject: Re: mmap() incoherency on hi I/O load (FS is zfs) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jul 2012 09:06:43 -0000 --vX2ve0hYIA9A871f Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jul 04, 2012 at 11:07:36AM +0300, Pavlo wrote: >=20 >=20 >=20 > --- Original message --- > From: "Pavlo" > To: freebsd-fs@freebsd.org > Date: 14 June 2012, 13:30:20 > Subject: mmap() incoherency on hi I/O load (FS is zfs) >=20 >=20 > > There's a case when some parts of files that are mapped and then > modified getting corrupted. By corrupted I mean some data is ok (one that > was written using write()/pwrite()) but some looks like it never existed. > Like it was some time in buffers, when several processes simultaneously > (of course access was synchronised) used shared pages and reported it's > existence. But after time pass they (processes) screamed that it is now > lost. Only part of data written with pwrite() was there. Everything that > was written via mmap() is zero. > > > > So as I said it occurs on hi I/O busyness. When in background 4+ > processes do indexing of huge ammount of data. Also I want to note, it > never occurred in the life of our project while we used mmap() under > same I/O stress conditions when mapping was done for a whole file of just > a part(header) starting from a beginning of a file. First time we used > mapping of individual pages, just to save RAM, and this popped up. > > > > Solution for this problem is msync() before any munmap(). But man says: > > > > >=20 > The msync() system call is usually not needed since BSD implements a > coherent file system buffer cache. However, it may be used to associate > dirty VM pages with file system buffers and thus cause them to be flushed > to physical media sooner rather than later. > >=20 > > Any thoughts? Thanks. > >=20 > >=20 >=20 > So I tracked issue to the place where it occurs. When I commit data to > file using mmap() and pwrite() side by side, sometimes 'newest data' is > being overwritten by 'elder data'. From time to time 'elder data' can be > something written with mmap() either with pwrite(). It never happens when > I use exclusively mmap() either pwrite(). Also this issue reproduces on > UFS as well. I think there is a problem keeping mmapep pages and FS cache > synced. I am curious how do you label data with newer and elder labels. I do admit a possibility of a race in ZFS double-copy implementation of the mmap/cache coherency, but somewhat skeptical about the same possibility for UFS. What you saying might indicate that we loose modified/dirty bits for the page, but that would have much more firework then just eventual race with write. What version of the system ? Does the machine swap ? >=20 > I will try to make test to reliably reproduces issue. Yes, isolated test case is the best route forward. It would either show a bug or demonstrate a misunderstanding on your part. --vX2ve0hYIA9A871f Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk/0B5kACgkQC3+MBN1Mb4gZ0QCg7SoPwIYcseI/gSsbOOyTboCN oxgAn0HWsYDFOdsxdsedeuEbucXyGPUc =DUC7 -----END PGP SIGNATURE----- --vX2ve0hYIA9A871f--