From owner-freebsd-fs@FreeBSD.ORG Thu Sep 25 07:05:00 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50FF869 for ; Thu, 25 Sep 2014 07:05:00 +0000 (UTC) Received: from relay01.pair.com (relay01.pair.com [209.68.5.15]) by mx1.freebsd.org (Postfix) with SMTP id 11E91CBF for ; Thu, 25 Sep 2014 07:04:59 +0000 (UTC) Received: (qmail 46722 invoked from network); 25 Sep 2014 07:04:58 -0000 Received: from 87.58.146.155 (HELO x2.osted.lan) (87.58.146.155) by relay01.pair.com with SMTP; 25 Sep 2014 07:04:58 -0000 X-pair-Authenticated: 87.58.146.155 Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.14.5/8.14.5) with ESMTP id s8P74uWO045229; Thu, 25 Sep 2014 09:04:56 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.5/8.14.5/Submit) id s8P74u90045222; Thu, 25 Sep 2014 09:04:56 +0200 (CEST) (envelope-from pho) Date: Thu, 25 Sep 2014 09:04:56 +0200 From: Peter Holm To: Konstantin Belousov Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140925070456.GA43144@x2.osted.lan> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <20140924134725.GI8870@kib.kiev.ua> <20140924153045.GA15685@x2.osted.lan> <20140924155728.GN8870@kib.kiev.ua> <20140924171509.GA18965@x2.osted.lan> <20140924174315.GO8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924174315.GO8870@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Sep 2014 07:05:00 -0000 On Wed, Sep 24, 2014 at 08:43:15PM +0300, Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 07:15:09PM +0200, Peter Holm wrote: > > On Wed, Sep 24, 2014 at 06:57:28PM +0300, Konstantin Belousov wrote: > > > On Wed, Sep 24, 2014 at 05:30:45PM +0200, Peter Holm wrote: > > > > On Wed, Sep 24, 2014 at 04:47:25PM +0300, Konstantin Belousov wrote: > > > > > On Wed, Sep 24, 2014 at 03:26:05PM +0200, Peter Holm wrote: > > > > > > The patch is an improvement, but: > > > > > > > > > > > > http://people.freebsd.org/~pho/stress/log/kostik718.txt > > > > > > > > > > Does you load included both rename and link, or only one of those > > > > > syscalls ? I see a bug in the rename part of the patch, below is > > > > > the update. > > > > > > > > > > > > > Both. I have split the tests in two now. Uptime is by now one hour. > > > > I'll let that run for a few hours more, before switching to random > > > > tests. > > > > > > > > I did get this page fault once: > > > > http://people.freebsd.org/~pho/stress/log/kostik719.txt > > > > but I guess it's unrelated? I have recompiled uma_core.c and > > > > vm_pageout with "-O0" in case it shows up again. > > > > > > This looks unrelated. But, in the log, I see user-controllable LOR > > > caused by my patch. Please use the following update instead. > > > > > > diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c > > > index b3b7ed5..a4aa19e 100644 > > > --- a/sys/kern/vfs_syscalls.c > > > > Seems unchanged to me? > > > > 20140924 19:08:31 all (1/2): link.sh > > lock order reversal: > > 1st 0xfffff800b06ce068 ufs (ufs) @ kern/vfs_subr.c:2137 > > 2nd 0xfffffe0785edfeb8 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 > > 3rd 0xfffff800b06a6548 ufs (ufs) @ kern/vfs_subr.c:2137 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19150 > > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19200 > > witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db19290 > > __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db193c0 > > ffs_lock() at ffs_lock+0x92/frame 0xfffffe081db19410 > > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19440 > > _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db194b0 > > vget() at vget+0x67/frame 0xfffffe081db194f0 > > vfs_hash_get() at vfs_hash_get+0xe1/frame 0xfffffe081db19540 > > ffs_vgetf() at ffs_vgetf+0x40/frame 0xfffffe081db195d0 > > softdep_sync_buf() at softdep_sync_buf+0xac0/frame 0xfffffe081db196b0 > > ffs_syncvnode() at ffs_syncvnode+0x286/frame 0xfffffe081db19730 > > ffs_sync() at ffs_sync+0x20f/frame 0xfffffe081db197f0 > > dounmount() at dounmount+0x3da/frame 0xfffffe081db19870 > > sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 > > amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 > > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 > > --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- > > lock order reversal: > > 1st 0xfffff800290fa068 ufs (ufs) @ kern/vfs_mount.c:1223 > > 2nd 0xfffff800b0214068 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1375 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19370 > > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19420 > > witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db194b0 > > __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db195e0 > > vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe081db19600 > > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19630 > > _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db196a0 > > ffs_flushfiles() at ffs_flushfiles+0x120/frame 0xfffffe081db19710 > > softdep_flushfiles() at softdep_flushfiles+0x232/frame 0xfffffe081db19780 > > ffs_unmount() at ffs_unmount+0xe5/frame 0xfffffe081db197f0 > > dounmount() at dounmount+0x424/frame 0xfffffe081db19870 > > sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 > > amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 > > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 > > --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- > > 20140924 19:10:34 all (2/2): link2.sh > > > > with > > FreeBSD 11.0-CURRENT (PHO) #0 r272060M: Wed Sep 24 19:00:20 CEST 2014 > > No, these two are known and harmless. The patch added new LOR, > where you link between two different mount points. The code first > locked vnodes, and only then checked for EXDEV. The log show some > like ufs/tmpfs etc. Yes thank you, I see. There seems to be a problem with the patch, causing a umount to fail: # umount /mnt umount: unmount of /mnt failed: Device busy # http://people.freebsd.org/~pho/stress/log/kostik720.txt -- Peter