From owner-freebsd-fs@freebsd.org Sun Jan 10 15:45:26 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD584A6909C for ; Sun, 10 Jan 2016 15:45:26 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 67E931D46 for ; Sun, 10 Jan 2016 15:45:26 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0AFjJ1P074173 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 10 Jan 2016 17:45:20 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0AFjJ1P074173 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0AFjIXQ074160; Sun, 10 Jan 2016 17:45:18 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 10 Jan 2016 17:45:18 +0200 From: Konstantin Belousov To: Rick Macklem Cc: FreeBSD Filesystems Subject: Re: panic ffs_truncate3 (maybe fuse being evil) Message-ID: <20160110154518.GU3625@kib.kiev.ua> References: <1696608910.154845456.1452438117036.JavaMail.zimbra@uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1696608910.154845456.1452438117036.JavaMail.zimbra@uoguelph.ca> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2016 15:45:26 -0000 On Sun, Jan 10, 2016 at 10:01:57AM -0500, Rick Macklem wrote: > Hi, > > When fooling around with GlusterFS, I can get this panic intermittently. > (I had a couple yesterday.) This happens on a Dec. 5, 2015 head kernel. > > panic: ffs_truncate3 > - backtrace without the numbers (I just scribbled it off the screen) > ffs_truncate() > ufs_inactive() > VOP_INACTIVE_APV() > vinactive() > vputx() > kern_unlinkat() > > So, at a glance, it seems that either > b_dirty.bv_cnt > or b_clean.bv_cnt > is non-zero. (There is another case for the panic, but I thought it > was less likely?) > > So, I'm wondering if this might be another side effect of r291460, > since after that a new vnode isn't completely zero'd out? > > However, shouldn't bo_dirty.bv_cnt and bo_clean.bv_cnt be zero when > a vnode is recycled? > Does this make sense or do some fields of v_bufobj need to be zero'd > out by getnewvnode()? Look at the _vdrop(). When a vnode is freed to zone, it is asserted that bufobj queues are empty. I very much doubt that it is possible to leak either buffers or counters by reuse. > > GlusterFS is using fuse and I suspect that fuse isn't cleaning out > the buffers under some circumstance (I already noticed that there > isn't any code in its fuse_vnop_reclaim() and I vaguely recall that > there are conditions where VOP_INACTIVE() gets skipped, so that VOP_RECLAIM() > has to check for anything that would have been done by VOP_INACTIVE() > and do it, if it isn't already done.) But even if fuse leaves the buffers around, is it UFS which panics for you ? I would rather worry about dandling pointers and use after free in fuse, which is a known issue with it anyway. I.e. it could be that fuse operates on reclaimed and reused vnode as its own. > > Anyhow, if others have thoughts on this (or other hunches w.r.t. what > could cause this panic(), please let me know. The ffs_truncate3 was deterministically triggered by a bug in ffs_balloc(). The routine allocated buffers for indirect blocks, but if the blocks cannot be allocated, the buffers where left on queue. See r174973, this was fixed very long time ago.