From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 17:15:12 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6E7CEEBA for ; Wed, 24 Sep 2014 17:15:12 +0000 (UTC) Received: from relay03.pair.com (relay03.pair.com [209.68.5.17]) by mx1.freebsd.org (Postfix) with SMTP id 31CD9D2C for ; Wed, 24 Sep 2014 17:15:11 +0000 (UTC) Received: (qmail 47440 invoked from network); 24 Sep 2014 17:15:10 -0000 Received: from 87.58.146.155 (HELO x2.osted.lan) (87.58.146.155) by relay03.pair.com with SMTP; 24 Sep 2014 17:15:10 -0000 X-pair-Authenticated: 87.58.146.155 Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.14.5/8.14.5) with ESMTP id s8OHF9lR019172; Wed, 24 Sep 2014 19:15:09 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.5/8.14.5/Submit) id s8OHF9aZ019171; Wed, 24 Sep 2014 19:15:09 +0200 (CEST) (envelope-from pho) Date: Wed, 24 Sep 2014 19:15:09 +0200 From: Peter Holm To: Konstantin Belousov Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924171509.GA18965@x2.osted.lan> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <20140924134725.GI8870@kib.kiev.ua> <20140924153045.GA15685@x2.osted.lan> <20140924155728.GN8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924155728.GN8870@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 17:15:12 -0000 On Wed, Sep 24, 2014 at 06:57:28PM +0300, Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 05:30:45PM +0200, Peter Holm wrote: > > On Wed, Sep 24, 2014 at 04:47:25PM +0300, Konstantin Belousov wrote: > > > On Wed, Sep 24, 2014 at 03:26:05PM +0200, Peter Holm wrote: > > > > The patch is an improvement, but: > > > > > > > > http://people.freebsd.org/~pho/stress/log/kostik718.txt > > > > > > Does you load included both rename and link, or only one of those > > > syscalls ? I see a bug in the rename part of the patch, below is > > > the update. > > > > > > > Both. I have split the tests in two now. Uptime is by now one hour. > > I'll let that run for a few hours more, before switching to random > > tests. > > > > I did get this page fault once: > > http://people.freebsd.org/~pho/stress/log/kostik719.txt > > but I guess it's unrelated? I have recompiled uma_core.c and > > vm_pageout with "-O0" in case it shows up again. > > This looks unrelated. But, in the log, I see user-controllable LOR > caused by my patch. Please use the following update instead. > > diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c > index b3b7ed5..a4aa19e 100644 > --- a/sys/kern/vfs_syscalls.c Seems unchanged to me? 20140924 19:08:31 all (1/2): link.sh lock order reversal: 1st 0xfffff800b06ce068 ufs (ufs) @ kern/vfs_subr.c:2137 2nd 0xfffffe0785edfeb8 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 3rd 0xfffff800b06a6548 ufs (ufs) @ kern/vfs_subr.c:2137 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19150 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19200 witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db19290 __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db193c0 ffs_lock() at ffs_lock+0x92/frame 0xfffffe081db19410 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19440 _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db194b0 vget() at vget+0x67/frame 0xfffffe081db194f0 vfs_hash_get() at vfs_hash_get+0xe1/frame 0xfffffe081db19540 ffs_vgetf() at ffs_vgetf+0x40/frame 0xfffffe081db195d0 softdep_sync_buf() at softdep_sync_buf+0xac0/frame 0xfffffe081db196b0 ffs_syncvnode() at ffs_syncvnode+0x286/frame 0xfffffe081db19730 ffs_sync() at ffs_sync+0x20f/frame 0xfffffe081db197f0 dounmount() at dounmount+0x3da/frame 0xfffffe081db19870 sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- lock order reversal: 1st 0xfffff800290fa068 ufs (ufs) @ kern/vfs_mount.c:1223 2nd 0xfffff800b0214068 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1375 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19370 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19420 witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db194b0 __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db195e0 vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe081db19600 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19630 _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db196a0 ffs_flushfiles() at ffs_flushfiles+0x120/frame 0xfffffe081db19710 softdep_flushfiles() at softdep_flushfiles+0x232/frame 0xfffffe081db19780 ffs_unmount() at ffs_unmount+0xe5/frame 0xfffffe081db197f0 dounmount() at dounmount+0x424/frame 0xfffffe081db19870 sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- 20140924 19:10:34 all (2/2): link2.sh with FreeBSD 11.0-CURRENT (PHO) #0 r272060M: Wed Sep 24 19:00:20 CEST 2014 - Peter