From owner-freebsd-stable@freebsd.org Sun Nov 13 10:07:24 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD7B9C3D749 for ; Sun, 13 Nov 2016 10:07:24 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citapm.icyb.net.ua (citapm.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id B3BD81ACA; Sun, 13 Nov 2016 10:07:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citapm.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA00119; Sun, 13 Nov 2016 12:07:21 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1c5rh3-000B87-DN; Sun, 13 Nov 2016 12:07:21 +0200 Subject: Re: Freebsd 11.0 RELEASE - ZFS deadlock To: Henri Hennebert , freebsd-stable@FreeBSD.org References: <0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> <43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org> <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> <6bc95dce-31e1-3013-bfe3-7c2dd80f9d1e@restart.be> <23a66749-f138-1f1a-afae-c775f906ff37@restart.be> <8e7547ef-87f7-7fab-6f45-221e8cea1989@FreeBSD.org> <6d991cea-b420-531e-12cc-001e4aeed66b@restart.be> <67f2e8bd-bff0-f808-7557-7dabe5cad78c@FreeBSD.org> <1cb09c54-5f0e-2259-a41a-fefe76b4fe8b@restart.be> <9f20020b-e2f1-862b-c3fc-dc6ff94e301e@restart.be> <599c5a5b-aa08-2030-34f3-23ff19d09a9b@restart.be> Cc: Konstantin Belousov From: Andriy Gapon Message-ID: <32686283-948a-6faf-7ded-ed8fcd23affb@FreeBSD.org> Date: Sun, 13 Nov 2016 12:06:00 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <599c5a5b-aa08-2030-34f3-23ff19d09a9b@restart.be> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2016 10:07:24 -0000 On 12/11/2016 14:40, Henri Hennebert wrote: > I attatch it Thank you! So, these two threads are trying to get the lock in the exclusive mode: Thread 687 (Thread 101243): #0 sched_switch (td=0xfffff800b642b500, newtd=0xfffff8000285ea00, flags=) at /usr/src/sys/kern/sched_ule.c:1973 #1 0xffffffff80561ae2 in mi_switch (flags=, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:455 #2 0xffffffff805ae8da in sleepq_wait (wchan=0x0, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:646 #3 0xffffffff8052f854 in sleeplk (lk=, flags=, ilk=, wmesg=0xffffffff813be535 "zfs", pri=, timo=51) at /usr/src/sys/kern/kern_lock.c:222 #4 0xffffffff8052f39d in __lockmgr_args (lk=, flags=, ilk=, wmesg=, pri=, timo=, file=, line=) at /usr/src/sys/kern/kern_lock.c:958 #5 0xffffffff80616a8c in vop_stdlock (ap=) at lockmgr.h:98 #6 0xffffffff8093784d in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2087 #7 0xffffffff8063c5b3 in _vn_lock (vp=, flags=548864, file=, line=) at vnode_if.h:859 #8 0xffffffff8062a5f7 in vget (vp=0xfffff80049c2c000, flags=548864, td=0xfffff800b642b500) at /usr/src/sys/kern/vfs_subr.c:2523 #9 0xffffffff806118b9 in cache_lookup (dvp=, vpp=, cnp=, tsp=, ticksp=) at /usr/src/sys/kern/vfs_cache.c:686 #10 0xffffffff806133dc in vfs_cache_lookup (ap=) at /usr/src/sys/kern/vfs_cache.c:1081 #11 0xffffffff80935777 in VOP_LOOKUP_APV (vop=, a=) at vnode_if.c:127 #12 0xffffffff8061cdf1 in lookup (ndp=) at vnode_if.h:54 #13 0xffffffff8061c492 in namei (ndp=) at /usr/src/sys/kern/vfs_lookup.c:306 #14 0xffffffff80509395 in kern_execve (td=, args=, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:443 #15 0xffffffff80508ccc in sys_execve (td=0xfffff800b642b500, uap=0xfffffe010182cb80) at /usr/src/sys/kern/kern_exec.c:218 #16 0xffffffff808d449e in amd64_syscall (td=, traced=0) at subr_syscall.c:135 #17 0xffffffff808b7ddb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 Thread 681 (Thread 101147): #0 sched_switch (td=0xfffff80065f4e500, newtd=0xfffff8000285f000, flags=) at /usr/src/sys/kern/sched_ule.c:1973 #1 0xffffffff80561ae2 in mi_switch (flags=, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:455 #2 0xffffffff805ae8da in sleepq_wait (wchan=0x0, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:646 #3 0xffffffff8052f854 in sleeplk (lk=, flags=, ilk=, wmesg=0xffffffff813be535 "zfs", pri=, timo=51) at /usr/src/sys/kern/kern_lock.c:222 #4 0xffffffff8052f39d in __lockmgr_args (lk=, flags=, ilk=, wmesg=, pri=, timo=, file=, line=) at /usr/src/sys/kern/kern_lock.c:958 #5 0xffffffff80616a8c in vop_stdlock (ap=) at lockmgr.h:98 #6 0xffffffff8093784d in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2087 #7 0xffffffff8063c5b3 in _vn_lock (vp=, flags=548864, file=, line=) at vnode_if.h:859 #8 0xffffffff8062a5f7 in vget (vp=0xfffff80049c2c000, flags=548864, td=0xfffff80065f4e500) at /usr/src/sys/kern/vfs_subr.c:2523 #9 0xffffffff806118b9 in cache_lookup (dvp=, vpp=, cnp=, tsp=, ticksp=) at /usr/src/sys/kern/vfs_cache.c:686 #10 0xffffffff806133dc in vfs_cache_lookup (ap=) at /usr/src/sys/kern/vfs_cache.c:1081 #11 0xffffffff80935777 in VOP_LOOKUP_APV (vop=, a=) at vnode_if.c:127 #12 0xffffffff8061cdf1 in lookup (ndp=) at vnode_if.h:54 #13 0xffffffff8061c492 in namei (ndp=) at /usr/src/sys/kern/vfs_lookup.c:306 #14 0xffffffff80509395 in kern_execve (td=, args=, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:443 #15 0xffffffff80508ccc in sys_execve (td=0xfffff80065f4e500, uap=0xfffffe01016b8b80) at /usr/src/sys/kern/kern_exec.c:218 #16 0xffffffff808d449e in amd64_syscall (td=, traced=0) at subr_syscall.c:135 #17 0xffffffff808b7ddb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 And the original stuck thread wants to get the lock in the shared mode. And there should be another thread that already holds the lock in the shared mode. But I am not able to identify it. I wonder if the original thread could be trying to get the lock recursively... It would be interesting to get more details from thread 101112. You can switch to it using tid command, you can use 'fr' to select frames, 'info local' and 'info args' to see what variables are available (not optimized out) and the you can print any that look interesting. It would be nice to get a file path and a directory vnode where the lookup is called. Thank you. -- Andriy Gapon