Date: Fri, 16 Mar 2007 08:33:24 +1000 From: David Cecil <david.cecil@nokia.com> To: ext Pawel Jakub Dawidek <pjd@FreeBSD.org> Cc: freebsd-fs@FreeBSD.org Subject: Re: FFS writes to read-only mount Message-ID: <45F9C9B4.4030508@nokia.com> In-Reply-To: <20070315092659.GA14080@garage.freebsd.pl> References: <45F776AE.8090702@nokia.com> <20070314161041.GI7847@garage.freebsd.pl> <45F8EE27.6070208@nokia.com> <20070315090031.GB80993@deviant.kiev.zoral.com.ua> <20070315092659.GA14080@garage.freebsd.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi guys, > On Thu, Mar 15, 2007 at 11:00:31AM +0200, Kostik Belousov wrote: > >> On Thu, Mar 15, 2007 at 04:56:39PM +1000, David Cecil wrote: >> >>> Hi Pawel, >>> >>> here is what I've found. Note that this problem doesn't occur >>> frequently, but does happen often enough to be a problem. I am still >>> debugging the live system. >>> >>> I put breakpoints in g_print_bio and g_io_request. I then continued >>> until the breakpoint at g_print_bio breakpoint was hit, but each time >>> g_io_request was hit, I printed the backtrace. Then, at g_print_bio I >>> could identify the struct bio * and look for the corresponding >>> g_io_request trace with a matching struct bio *. >>> >>> That trace looks like this: >>> db> bt >>> Tracing pid 47 tid 100050 td 0x860d0640 >>> g_io_request(864258c4,860ec440,a0020024,98b92188,a5281bf0,...) at >>> g_io_request+0x107 >>> g_vfs_strategy(86104ce4,98b92188,a0020024,98b92188,86104c30,...) at >>> g_vfs_strategy+0x63 >>> ffs_geom_strategy(86104ce4,98b92188) at ffs_geom_strategy+0x129 >>> bufwrite(98b92188,4000,98b92188,a5281c54,805b20c1,...) at bufwrite+0x146 >>> ffs_bufwrite(98b92188) at ffs_bufwrite+0x282 >>> vfs_bio_awrite(98b92188) at vfs_bio_awrite+0x221 >>> vop_stdfsync(a5281cbc,a5281cbc,a5281c98,806eff3e,a5281cbc,...) at >>> vop_stdfsync+0x121 >>> >> It would be helpful, at this frame, to print the actual vnode, as well as >> the dirty buffer header together with dump of buffer data. [...] >> > > This comes from devfs_fsync(), so I suspect it's devvp, and we are > synchronizing entire file system here. > I think my interpretation corresponds with yours. The loop in vop_stdfsync is syncing the dirty buffers associated with the vnode that maps to /dev/mirror/gm0s1a >> [...] It seems that >> the easiest way to do that would be to obtain coredump and then use kgdb. >> > > On successfull remount to read-only we could walk (under DIAGNOSTIC) the > list of vnodes and assert there are no dirty buffers, but I don't think > this is the case... > I am still debugging the system in ddb, so I assume I could do that now. > The thing that worries me is that it doesn't happen from what I > understand just after remount to read-only. Syncer wakeups every 30 > seconds by default, right? Yes. > So it should generates errors on first time > he synchronizes file system. Which bascially means, that something tries > to write to read-only file system later. Will be good to assert it > somehow earlier. One thing that comes to my mind is to assert: > > KASSERT(((struct g_consumer *)buf->b_bufobj.bo_private)->acw > 0); > > in each b*write() function. This is probably not the only way to write > to a file system... > Okay, that's something for me to bear in mind. I as trying to get as much from the ddb session as I can, before rebooting and trying kernel mods. This problem occurs frequently enough to be annoying, but not so frequently that it's easy to reproduce. > I still don't feel too familiar with the code, so not sure if my > comments are usable:) how-to-reproduce procedure would be the best of > course... David, do you use something which can be considered > non-standard, like snapshots? > It may be that snapshots are used, but not explicitly. The startup scripts attempt to run fsck in the background, which would normally require a snapshot, but shouldn't for a read-only mount, right? I can't think of anything else that's different to standard in this area (FFS/GEOM)... Regards, Dave -- Software Engineer Secure and Mobile Connectivity Nokia Enterprise Solutions +61 7 5553 8307 (office) +61 412 728 222 (cell)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?45F9C9B4.4030508>