From owner-freebsd-fs@freebsd.org Tue Feb 2 23:19:30 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC243A992F8 for ; Tue, 2 Feb 2016 23:19:30 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 912151208 for ; Tue, 2 Feb 2016 23:19:30 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:xDmrVRU/YtgNq6KrQhn4qEVVtR3V8LGtZVwlr6E/grcLSJyIuqrYZhyFt8tkgFKBZ4jH8fUM07OQ6PC/HzVcqs/b+Fk5M7VyFDY9wf0MmAIhBMPXQWbaF9XNKxIAIcJZSVV+9Gu6O0UGUOz3ZlnVv2HgpWVKQka3CwN5K6zPF5LIiIzvjqbpq8KVOlkD3WD1SIgxBSv1hD2ZjtMRj4pmJ/R54TryiVwMRd5rw3h1L0mYhRf265T41pdi9yNNp6BprJYYAu3SNp41Rr1ADTkgL3t9pIiy7UGCHkOz4S4/VWMNjhNOHwnDpDv3WpDsqSzk/r5+3zKGPM78QLQcVjGr7qMtQxjt3nQpLTk8pVvWgc84qatQoxasolQr2Yvda4KROf9WY6TSYN4eXWoHVc8HBH8JOZ+1c4ZaV7lJBu1ftYSo4gJW9RY= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CrBACrOLFW/61jaINehH+IU7Nehg0CggIQAQEBAQEBAQGBCYItghQBAQEDASNWBQsCAQgOCgICDRkCAlcCBIgmCLBCjnoBAQEBAQUBAQEBAQEae4UUgXWCQoQCEQEKgxSBOgWNJXSIWIsWhBAWhCyDJoUuRI15AjcrggMZgWYeiGk0fAEBAQ X-IronPort-AV: E=Sophos;i="5.22,386,1449550800"; d="scan'208";a="265943699" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 02 Feb 2016 18:19:29 -0500 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 6152015F578; Tue, 2 Feb 2016 18:19:29 -0500 (EST) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id SR7FQ4djlWS2; Tue, 2 Feb 2016 18:19:29 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id E987A15F57B; Tue, 2 Feb 2016 18:19:28 -0500 (EST) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id G0PlJE4WIzfg; Tue, 2 Feb 2016 18:19:28 -0500 (EST) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id D190E15F578; Tue, 2 Feb 2016 18:19:28 -0500 (EST) Date: Tue, 2 Feb 2016 18:19:28 -0500 (EST) From: Rick Macklem To: Konstantin Belousov Cc: FreeBSD Filesystems , Kirk McKusick Message-ID: <1548616575.1027329.1454455168829.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <20160202144812.GZ91220@kib.kiev.ua> References: <1696608910.154845456.1452438117036.JavaMail.zimbra@uoguelph.ca> <20160116191116.GI3942@kib.kiev.ua> <853868666.163292727.1452986431921.JavaMail.zimbra@uoguelph.ca> <20160117035858.GO3942@kib.kiev.ua> <855760730.165482737.1453177516248.JavaMail.zimbra@uoguelph.ca> <20160201221710.GR91220@kib.kiev.ua> <1375939202.186412561.1454375239581.JavaMail.zimbra@uoguelph.ca> <20160202144812.GZ91220@kib.kiev.ua> Subject: Re: panic ffs_truncate3 (maybe fuse being evil) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF44 (Win)/8.0.9_GA_6191) Thread-Topic: panic ffs_truncate3 (maybe fuse being evil) Thread-Index: /iMtGWGw+IHAv5MnC9gdab2CSHP59w== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Feb 2016 23:19:31 -0000 Kostik wrote: > On Mon, Feb 01, 2016 at 08:07:19PM -0500, Rick Macklem wrote: > > Kostik wrote: > > > On Mon, Jan 18, 2016 at 11:25:16PM -0500, Rick Macklem wrote: > > > > Kostik wrote: > > > > > On Sat, Jan 16, 2016 at 06:20:31PM -0500, Rick Macklem wrote: > > > > > > Kostik wrote: > > > > > > > Was IO_EXT flag passed to the ffs_truncate() invocation where the > > > > > > > assert ffs_truncate3 fired ? > > > > > > > > > > > > > Yes. The only call to UFS_TRUNCATE() in ufs_inactive() specified > > > > > > both > > > > > > IO_EXT | IO_NORMAL. > > > > > > > > > > Please try this. > > > > > > > > > Still happens. I put the old panic test in, but with a printf() instead > > > > of panic() and the printf() happens with this patch. > > > > Btw, whenever I've looked, the entry is on the clean list. > > > > No other panics happen and the file system fsck's fine after the test > > > > run. > > > > > > I thought that you posted a stand-alone program which triggered the issue > > > on non-SU mounts, but I cannot find it. Does such program exist, or this > > > is just me misremembering ? > > > > > Nope. I do a test run using GlusterFS (a FreeBSD kernel build with the > > sources > > on GlusterFS) where the bricks are on UFS file systems with SU disabled. > > > > To be honest, I've been using ZFS for tests lately, so I'm not even sure I > > remember exactly what the test setup was, but I think the above is it. > > (It takes 2-3hrs for an occurrence to happen.) > > > > I did plan on taking a look at vinvalbuf(..V_ALT..), since that is what I > > think should be getting rid of the extended attribute block, but haven't > > done so. > I doubt that this is vinvalbuf(), more likely it is stray buffer forgotten > by some cleanup code. Note that ffs_truncate() condition to call > vinvalbuf(V_ALT) is that IO_EXT flag is passed and dinode di_extsize field > value is > 0. > I didn't notice the di_extsize > 0 and I see the panic is with di_extsize == 0. (At a glance I didn't see anything suspicious in vinvalbuf() anyhow.) > > > > If you have another patch to try, email it and I'll try and reproduce the > > failure without/with the patch. > I asked about the test program to be able to trace the issue. Unfortunately I doubt a simple program will reproduce it. GlusterFS loves to read extended attributes, so I'd guess it has read at least 100,000 of them by the time it hits the panic in a test run. (It loves doing system calls, so I don't run ktrace for long, but I recall 10+ get_link_extattr (or whatever the exact name is, the "get extended attribute without following symlinks" one) in a 1second ktrace.) rick > I do think that the IO_UNIT patch fixes some situations where the buffer > can be left on the vnode queue. I did not found any other places so far. >