From owner-freebsd-fs@FreeBSD.ORG Thu Mar 15 09:27:14 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A526D16A400 for ; Thu, 15 Mar 2007 09:27:14 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 0154F13C45A for ; Thu, 15 Mar 2007 09:27:13 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 8A51A456B1; Thu, 15 Mar 2007 10:27:12 +0100 (CET) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id C058745681; Thu, 15 Mar 2007 10:27:04 +0100 (CET) Date: Thu, 15 Mar 2007 10:26:59 +0100 From: Pawel Jakub Dawidek To: Kostik Belousov Message-ID: <20070315092659.GA14080@garage.freebsd.pl> References: <45F776AE.8090702@nokia.com> <20070314161041.GI7847@garage.freebsd.pl> <45F8EE27.6070208@nokia.com> <20070315090031.GB80993@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="cWoXeonUoKmBZSoM" Content-Disposition: inline In-Reply-To: <20070315090031.GB80993@deviant.kiev.zoral.com.ua> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: FFS writes to read-only mount X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Mar 2007 09:27:14 -0000 --cWoXeonUoKmBZSoM Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 15, 2007 at 11:00:31AM +0200, Kostik Belousov wrote: > On Thu, Mar 15, 2007 at 04:56:39PM +1000, David Cecil wrote: > > Hi Pawel, > >=20 > > here is what I've found. Note that this problem doesn't occur=20 > > frequently, but does happen often enough to be a problem. I am still= =20 > > debugging the live system. > >=20 > > I put breakpoints in g_print_bio and g_io_request. I then continued=20 > > until the breakpoint at g_print_bio breakpoint was hit, but each time= =20 > > g_io_request was hit, I printed the backtrace. Then, at g_print_bio I= =20 > > could identify the struct bio * and look for the corresponding=20 > > g_io_request trace with a matching struct bio *. > >=20 > > That trace looks like this: > > db> bt > > Tracing pid 47 tid 100050 td 0x860d0640 > > g_io_request(864258c4,860ec440,a0020024,98b92188,a5281bf0,...) at=20 > > g_io_request+0x107 > > g_vfs_strategy(86104ce4,98b92188,a0020024,98b92188,86104c30,...) at=20 > > g_vfs_strategy+0x63 > > ffs_geom_strategy(86104ce4,98b92188) at ffs_geom_strategy+0x129 > > bufwrite(98b92188,4000,98b92188,a5281c54,805b20c1,...) at bufwrite+0x146 > > ffs_bufwrite(98b92188) at ffs_bufwrite+0x282 > > vfs_bio_awrite(98b92188) at vfs_bio_awrite+0x221 > > vop_stdfsync(a5281cbc,a5281cbc,a5281c98,806eff3e,a5281cbc,...) at=20 > > vop_stdfsync+0x121 > It would be helpful, at this frame, to print the actual vnode, as well as > the dirty buffer header together with dump of buffer data. [...] This comes from devfs_fsync(), so I suspect it's devvp, and we are synchronizing entire file system here. > [...] It seems that > the easiest way to do that would be to obtain coredump and then use kgdb. On successfull remount to read-only we could walk (under DIAGNOSTIC) the list of vnodes and assert there are no dirty buffers, but I don't think this is the case... The thing that worries me is that it doesn't happen from what I understand just after remount to read-only. Syncer wakeups every 30 seconds by default, right? So it should generates errors on first time he synchronizes file system. Which bascially means, that something tries to write to read-only file system later. Will be good to assert it somehow earlier. One thing that comes to my mind is to assert: KASSERT(((struct g_consumer *)buf->b_bufobj.bo_private)->acw > 0); in each b*write() function. This is probably not the only way to write to a file system... I still don't feel too familiar with the code, so not sure if my comments are usable:) how-to-reproduce procedure would be the best of course... David, do you use something which can be considered non-standard, like snapshots? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --cWoXeonUoKmBZSoM Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF+RFjForvXbEpPzQRAvYiAJ0ZU4xcQbTA3uh4g/DGPrWoVdiHLgCgmK1y hWJIJkQGGSwy3KHE4zlnPCY= =Wcdo -----END PGP SIGNATURE----- --cWoXeonUoKmBZSoM--