Date: Thu, 19 Mar 2009 12:46:05 -0400 From: John Baldwin <jhb@freebsd.org> To: Kostik Belousov <kostikbel@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: in recent 7-STABLE: VOP_WRITE...is not exclusive locked but should be Message-ID: <200903191246.05641.jhb@freebsd.org> In-Reply-To: <20090319160251.GJ7716@deviant.kiev.zoral.com.ua> References: <alpine.LFD.2.00.0903190639450.9228@cardinal> <200903191001.44491.jhb@freebsd.org> <20090319160251.GJ7716@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday 19 March 2009 12:02:51 pm Kostik Belousov wrote:
> On Thu, Mar 19, 2009 at 10:01:44AM -0400, John Baldwin wrote:
> > On Thursday 19 March 2009 8:05:34 am Tim Chase wrote:
> > > Hello,
> > >
> > > I have a system that had been running quite well with an oldish 7-STABLE
> > > (from around August 7, 2008) but has started deadlocking within the past
> > > week or so.
> > >
> > > I updated the kernel to a newer 7-STABLE (Mar 15, 2009) and enabled
> > > INVARIANTS, INVARIANT_SUPPORT, WITNESS, DEBUG_LOCKS DEBUG_VFS_LOCKS and
> > > DIAGNOSTIC and the message indicated in the subject line has now
appeared
> > > 3 times as shown below. Is this something to be terribly concerned
about?
> > > Is there anything I can to to further track down the cause? Since the
> > > system is a production mail server, I have it set to not drop into DDB
> > > when this happens.
> > >
> > > The machine is a 4-core Xeon X5450 with 8G of RAM running FreeBSD
> > > amd64 and in userland it's pretty much just cyrus imapd and apache/php.
> > > The file systems are all ZFS on a bunch of SAS drives connected to a
> > > LSI Logic 1068 controller.
> > >
> > > As to the deadlock that started this exercise, if the machine follows
its
> > > recent pattern, that should happen within the next 2-4 hours.
> >
> > Err, the vn_write() routine should be using an exclusive vnode lock:
> >
> > vn_write()
> > {
> >
> > ...
> > vn_lock(vp, LK_EXCLUSIVE | LK_RETRY, td);
> > if ((flags & FOF_OFFSET) == 0)
> > uio->uio_offset = fp->f_offset;
> > ioflag |= sequential_heuristic(uio, fp);
> > #ifdef MAC
> > error = mac_check_vnode_write(active_cred, fp->f_cred, vp);
> > if (error == 0)
> > #endif
> > error = VOP_WRITE(vp, uio, ioflag, fp->f_cred);
> > ...
> > }
> >
> > Can you check your /sys/kern/vfs_vnops.c and verify that LK_EXCLUSIVE is
> > present in your vn_write() routine? If so, then perhaps run memtest?
>
> Note that this happens on the ZFS. Might be, ZFS unlocks the vnode and
> then relocks it with some invalid flags ?
Hmm, it depends on where the check is I guess (before or after zfs_write()).
Tim, can you do 'l *VOP_WRITE_APV+0x155' on your kernel from gdb?
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200903191246.05641.jhb>