From owner-freebsd-stable@freebsd.org Thu Nov 10 17:55:29 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1BC8C3A479 for ; Thu, 10 Nov 2016 17:55:29 +0000 (UTC) (envelope-from hlh@restart.be) Received: from tignes.restart.be (tignes.restart.be [IPv6:2001:41d0:8:bdbe:0:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tignes.restart.be", Issuer "CA master" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 590D79BE; Thu, 10 Nov 2016 17:55:29 +0000 (UTC) (envelope-from hlh@restart.be) X-Comment: SPF check N/A for local connections - client-ip=2001:41d0:8:bdbe:1:1::; helo=restart.be; envelope-from=hlh@restart.be; receiver=avg@freebsd.org DKIM-Filter: OpenDKIM Filter v2.10.3 tignes.restart.be 3tF9gg2nP0zsct DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=restart.be; s=tignes; t=1478800527; bh=pyOsS6bkbIw8AgbtlxYlI6b+6cK6PXLWkO5FNE2yyn4=; h=Subject:To:References:From:Date:In-Reply-To; z=Subject:=20Re:=20Freebsd=2011.0=20RELEASE=20-=20ZFS=20deadlock|To :=20Andriy=20Gapon=20,=20freebsd-stable@FreeBSD.o rg|References:=20<0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> =0D=0A=20=0D=0A= 20<43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org>=0D=0A=20=0D=0A=20<9d1f9a76-5a8 d-6eca-9a50-907d55099847@FreeBSD.org>=0D=0A=20<6bc95dce-31e1-3013- bfe3-7c2dd80f9d1e@restart.be>=0D=0A=20=0D=0A=20<23a66749-f138-1f1a-afae-c775f906ff 37@restart.be>=0D=0A=20<8e7547ef-87f7-7fab-6f45-221e8cea1989@FreeB SD.org>|From:=20Henri=20Hennebert=20|Date:=20Thu,= 2010=20Nov=202016=2018:55:25=20+0100|In-Reply-To:=20<8e7547ef-87f7 -7fab-6f45-221e8cea1989@FreeBSD.org>; b=Mof5RdkDYRWSlDA4etFIunhKtZeivw9WGt7n5UiIlyKFYkPCcdINGLsayynb4ZZPt c+JEPr66wqEnRJo+fKOfdRBZ014Oweia6RleQdGVKiXGvSUt9C/KjgWKdU2oE+IxDI 5dFTy7sGPSmL3ZcVvdzRbsZ4B+ZB88zFCq1tLrcxPbs/Ep1eAcXP5CTKMDTxNtOFt4 J+xMfFdhtsjo5HTPA+4IewfFwEzCdhqyA6TOtbGQF4yCP/ndFqPltZTHQj17A9z3y9 kQIwmzFjpiHxycQ1VwIZOaw9dihkQVhWLZzdta+PCFODxygcaqJbkun4Pe8wUp7TmD LjrXwSBWFQ8EQ== Received: from restart.be (avoriaz.restart.be [IPv6:2001:41d0:8:bdbe:1:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.restart.be", Issuer "CA master" (verified OK)) by tignes.restart.be (Postfix) with ESMTPS id 3tF9gg2nP0zsct; Thu, 10 Nov 2016 18:55:26 +0100 (CET) Received: from chamonix.restart.bel (chamonix.restart.bel [IPv6:2001:41d0:8:bdbe:1:9:0:0]) (authenticated bits=0) by restart.be (8.15.2/8.15.2) with ESMTPSA id uAAHtP6l040043 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 10 Nov 2016 18:55:26 +0100 (CET) (envelope-from hlh@restart.be) Subject: Re: Freebsd 11.0 RELEASE - ZFS deadlock To: Andriy Gapon , freebsd-stable@FreeBSD.org References: <0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> <43c9d4d4-1995-5626-d70a-f92a5b456629@FreeBSD.org> <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> <6bc95dce-31e1-3013-bfe3-7c2dd80f9d1e@restart.be> <23a66749-f138-1f1a-afae-c775f906ff37@restart.be> <8e7547ef-87f7-7fab-6f45-221e8cea1989@FreeBSD.org> From: Henri Hennebert Message-ID: <6d991cea-b420-531e-12cc-001e4aeed66b@restart.be> Date: Thu, 10 Nov 2016 18:55:25 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <8e7547ef-87f7-7fab-6f45-221e8cea1989@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2016 17:55:29 -0000 On 11/10/2016 18:33, Andriy Gapon wrote: > On 10/11/2016 18:12, Henri Hennebert wrote: >> On 11/10/2016 16:54, Andriy Gapon wrote: >>> On 10/11/2016 17:20, Henri Hennebert wrote: >>>> On 11/10/2016 15:00, Andriy Gapon wrote: >>>>> Interesting. I can not spot any suspicious thread that would hold the vnode >>>>> lock. Could you please run kgdb (just like that, no arguments), then execute >>>>> 'bt' command and then select a frame when _vn_lock is called with 'fr N' >>>>> command. Then please 'print *vp' and share the result. >>>>> >>>> I Think I miss something in your request: >>> >>> Oh, sorry! The very first step should be 'tid 101112' to switch to the correct >>> context. >>> >> >> (kgdb) fr 7 >> #7 0xffffffff8063c5b3 in _vn_lock (vp=, flags=2121728, > > "value optimized out" - not good > >> file=, >> line=) at vnode_if.h:859 >> 859 vnode_if.h: No such file or directory. >> in vnode_if.h >> (kgdb) print *vp > > I am not sure if this output is valid, because of the message above. > Could you please try to navigate to nearby frames and see if vp itself has a > valid value there. If you can find such a frame please do *vp there. > Does this seems better? (kgdb) fr 8 #8 0xffffffff8062a5f7 in vget (vp=0xfffff80049c2c000, flags=2121728, td=0xfffff80009ba0500) at /usr/src/sys/kern/vfs_subr.c:2523 2523 if ((error = vn_lock(vp, flags)) != 0) { (kgdb) print *vp $1 = {v_tag = 0xffffffff813be535 "zfs", v_op = 0xffffffff813d0f70, v_data = 0xfffff80049c1f420, v_mount = 0xfffff800093aa660, v_nmntvnodes = {tqe_next = 0xfffff80049c2c938, tqe_prev = 0xfffff80049c2bb30}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = { tqh_first = 0xfffff800bfc8e3f0, tqh_last = 0xfffff800bfc8e410}, v_cache_dd = 0x0, v_lock = {lock_object = { lo_name = 0xffffffff813be535 "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0x0}, lk_lock = 23, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name = 0xffffffff8099e9e0 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, v_vnlock = 0xfffff80049c2c068, v_actfreelist = { tqe_next = 0xfffff80049c2c938, tqe_prev = 0xfffff80049ae9bd0}, v_bufobj = {bo_lock = {lock_object = { lo_name = 0xffffffff8099e9f0 "bufobj interlock", lo_flags = 86179840, lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, bo_ops = 0xffffffff80c4bf70, bo_object = 0xfffff800b62e9c60, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff80049c2c000, __bo_vnode = 0xfffff80049c2c000, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80049c2c120}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80049c2c140}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_bsize = 131072}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff80049c2c188}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, v_holdcnt = 9, v_usecount = 6, v_iflag = 512, v_vflag = 32, v_writecount = 0, v_hash = 4833984, v_type = VREG} (kgdb) Henri