Date: Sun, 10 Jan 2016 18:19:55 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Konstantin Belousov <kostikbel@gmail.com> Cc: FreeBSD Filesystems <freebsd-fs@freebsd.org> Subject: Re: panic ffs_truncate3 (maybe fuse being evil) Message-ID: <700310221.155153995.1452467995252.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20160110154518.GU3625@kib.kiev.ua> References: <1696608910.154845456.1452438117036.JavaMail.zimbra@uoguelph.ca> <20160110154518.GU3625@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
Kostik wrote: > On Sun, Jan 10, 2016 at 10:01:57AM -0500, Rick Macklem wrote: > > Hi, > > > > When fooling around with GlusterFS, I can get this panic intermittently. > > (I had a couple yesterday.) This happens on a Dec. 5, 2015 head kernel. > > > > panic: ffs_truncate3 > > - backtrace without the numbers (I just scribbled it off the screen) > > ffs_truncate() > > ufs_inactive() > > VOP_INACTIVE_APV() > > vinactive() > > vputx() > > kern_unlinkat() > > > > So, at a glance, it seems that either > > b_dirty.bv_cnt > > or b_clean.bv_cnt > > is non-zero. (There is another case for the panic, but I thought it > > was less likely?) > > > > So, I'm wondering if this might be another side effect of r291460, > > since after that a new vnode isn't completely zero'd out? > > > > However, shouldn't bo_dirty.bv_cnt and bo_clean.bv_cnt be zero when > > a vnode is recycled? > > Does this make sense or do some fields of v_bufobj need to be zero'd > > out by getnewvnode()? > Look at the _vdrop(). When a vnode is freed to zone, it is asserted > that bufobj queues are empty. I very much doubt that it is possible > to leak either buffers or counters by reuse. > Ok. I'll take a look but, yes, it doesn't sound like the fields could be left bogus when the vnode gets recycled. > > > > GlusterFS is using fuse and I suspect that fuse isn't cleaning out > > the buffers under some circumstance (I already noticed that there > > isn't any code in its fuse_vnop_reclaim() and I vaguely recall that > > there are conditions where VOP_INACTIVE() gets skipped, so that > > VOP_RECLAIM() > > has to check for anything that would have been done by VOP_INACTIVE() > > and do it, if it isn't already done.) > But even if fuse leaves the buffers around, is it UFS which panics for > you ? I would rather worry about dandling pointers and use after free in > fuse, which is a known issue with it anyway. I.e. it could be that fuse > operates on reclaimed and reused vnode as its own. > > > > > Anyhow, if others have thoughts on this (or other hunches w.r.t. what > > could cause this panic(), please let me know. > > The ffs_truncate3 was deterministically triggered by a bug in ffs_balloc(). > The routine allocated buffers for indirect blocks, but if the blocks cannot > be allocated, the buffers where left on queue. See r174973, this was fixed > very long time ago. > But this was a one month old kernel (around r291900, although I don't know the exact r#, but it was Dec. 5, 2015), so it definitely has this fix in it. When I see it again, I will try and see what the v_bufobj fields look like. Thanks, rick
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?700310221.155153995.1452467995252.JavaMail.zimbra>