From owner-freebsd-stable@freebsd.org Mon Nov 14 12:01:01 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69BEFC40263 for ; Mon, 14 Nov 2016 12:01:01 +0000 (UTC) (envelope-from hlh@restart.be) Received: from tignes.restart.be (tignes.restart.be [IPv6:2001:41d0:8:bdbe:0:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tignes.restart.be", Issuer "CA master" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8899814A3; Mon, 14 Nov 2016 12:01:00 +0000 (UTC) (envelope-from hlh@restart.be) X-Comment: SPF check N/A for local connections - client-ip=2001:41d0:8:bdbe:1:1::; helo=restart.be; envelope-from=hlh@restart.be; receiver=avg@freebsd.org DKIM-Filter: OpenDKIM Filter v2.10.3 tignes.restart.be 3tHTcp5zKczrdg DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=restart.be; s=tignes; t=1479124858; bh=huVPyPuglABUTT8SnByPhsXlXFzpkJeVpdYI84EI128=; h=Subject:To:References:Cc:From:Date:In-Reply-To; z=Subject:=20Re:=20Freebsd=2011.0=20RELEASE=20-=20ZFS=20deadlock|To :=20Andriy=20Gapon=20,=20freebsd-stable@FreeBSD.o rg|References:=20<0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> =0D=0A=20=0D=0A=2 0<9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org>=0D=0A=20<6bc95 dce-31e1-3013-bfe3-7c2dd80f9d1e@restart.be>=0D=0A=20=0D=0A=20<23a66749-f138-1f1a-a fae-c775f906ff37@restart.be>=0D=0A=20<8e7547ef-87f7-7fab-6f45-221e 8cea1989@FreeBSD.org>=0D=0A=20<6d991cea-b420-531e-12cc-001e4aeed66 b@restart.be>=0D=0A=20<67f2e8bd-bff0-f808-7557-7dabe5cad78c@FreeBS D.org>=0D=0A=20<1cb09c54-5f0e-2259-a41a-fefe76b4fe8b@restart.be>=0 D=0A=20=0D=0A=20 <9f20020b-e2f1-862b-c3fc-dc6ff94e301e@restart.be>=0D=0A=20=0D=0A=20<599c5a5b-aa08- 2030-34f3-23ff19d09a9b@restart.be>=0D=0A=20<32686283-948a-6faf-7de d-ed8fcd23affb@FreeBSD.org>=0D=0A=20=0D=0A=20=0D=0A=20<26512d69-94c2-92da-e3ea-50aebf17e3a0@restart .be>=0D=0A=20|Cc :=20Konstantin=20Belousov=20|From:=20Henri=20Henn ebert=20|Date:=20Mon,=2014=20Nov=202016=2013:00:57 =20+0100|In-Reply-To:=20; b=18Zc48vZxB1BQnjLEh1/RV4KKxi/KpZz0v+jr6VIwHMb9jqGqFFuV+M7R0ODaFgLl DAmDcKThRMpMKJEGXVWS0uaTpIg7aY//MsU6vX0vgZ/JvhIdUzYh7jn2ez3UPt8L5E dhboXHa4ACgiIBnETvOtNDtRKzjKazyaoT/g1RXxTAK8KD4Y8piok6XfzPhTn4ItZG zXNdyhPkgPz2/okOv79CFMOyP6bway/SHoxOf9VmIfREsBi7cOypDQl+pj2jXFfAbZ TGKLOr7UOujWnH7TKzzcPkToRJRuc9lugNCV5+BUlldynG3mIwcHWabn5+5tmdnEFS +CZ1nv1oWuIeA== Received: from restart.be (avoriaz.restart.be [IPv6:2001:41d0:8:bdbe:1:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.restart.be", Issuer "CA master" (verified OK)) by tignes.restart.be (Postfix) with ESMTPS id 3tHTcp5zKczrdg; Mon, 14 Nov 2016 13:00:58 +0100 (CET) Received: from chamonix.restart.bel (chamonix.restart.bel [IPv6:2001:41d0:8:bdbe:1:9:0:0]) (authenticated bits=0) by restart.be (8.15.2/8.15.2) with ESMTPSA id uAEC0vki001592 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 14 Nov 2016 13:00:57 +0100 (CET) (envelope-from hlh@restart.be) Subject: Re: Freebsd 11.0 RELEASE - ZFS deadlock To: Andriy Gapon , freebsd-stable@FreeBSD.org References: <0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> <9d1f9a76-5a8d-6eca-9a50-907d55099847@FreeBSD.org> <6bc95dce-31e1-3013-bfe3-7c2dd80f9d1e@restart.be> <23a66749-f138-1f1a-afae-c775f906ff37@restart.be> <8e7547ef-87f7-7fab-6f45-221e8cea1989@FreeBSD.org> <6d991cea-b420-531e-12cc-001e4aeed66b@restart.be> <67f2e8bd-bff0-f808-7557-7dabe5cad78c@FreeBSD.org> <1cb09c54-5f0e-2259-a41a-fefe76b4fe8b@restart.be> <9f20020b-e2f1-862b-c3fc-dc6ff94e301e@restart.be> <599c5a5b-aa08-2030-34f3-23ff19d09a9b@restart.be> <32686283-948a-6faf-7ded-ed8fcd23affb@FreeBSD.org> <26512d69-94c2-92da-e3ea-50aebf17e3a0@restart.be> Cc: Konstantin Belousov From: Henri Hennebert Message-ID: <80f65c86-1015-c409-1bf6-c01a5fe569c8@restart.be> Date: Mon, 14 Nov 2016 13:00:57 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2016 12:01:01 -0000 On 11/14/2016 12:45, Andriy Gapon wrote: > On 14/11/2016 11:35, Henri Hennebert wrote: >> >> >> On 11/14/2016 10:07, Andriy Gapon wrote: >>> Hmm, I've just noticed another interesting thread: >>> Thread 668 (Thread 101245): >>> #0 sched_switch (td=0xfffff800b642aa00, newtd=0xfffff8000285f000, flags=>> optimized out>) at /usr/src/sys/kern/sched_ule.c:1973 >>> #1 0xffffffff80561ae2 in mi_switch (flags=, newtd=0x0) at >>> /usr/src/sys/kern/kern_synch.c:455 >>> #2 0xffffffff805ae8da in sleepq_wait (wchan=0x0, pri=0) at >>> /usr/src/sys/kern/subr_sleepqueue.c:646 >>> #3 0xffffffff805614b1 in _sleep (ident=, lock=>> optimized out>, priority=, wmesg=0xffffffff809c51bc >>> "vmpfw", sbt=0, pr=, flags=) at >>> /usr/src/sys/kern/kern_synch.c:229 >>> #4 0xffffffff8089d1c1 in vm_page_busy_sleep (m=0xfffff800df68cd40, wmesg=>> optimized out>) at /usr/src/sys/vm/vm_page.c:753 >>> #5 0xffffffff8089dd4d in vm_page_sleep_if_busy (m=0xfffff800df68cd40, >>> msg=0xffffffff809c51bc "vmpfw") at /usr/src/sys/vm/vm_page.c:1086 >>> #6 0xffffffff80886be9 in vm_fault_hold (map=, vaddr=>> optimized out>, fault_type=4 '\004', fault_flags=0, m_hold=0x0) at >>> /usr/src/sys/vm/vm_fault.c:495 >>> #7 0xffffffff80885448 in vm_fault (map=0xfffff80011d66000, vaddr=>> optimized out>, fault_type=4 '\004', fault_flags=) at >>> /usr/src/sys/vm/vm_fault.c:273 >>> #8 0xffffffff808d3c49 in trap_pfault (frame=0xfffffe0101836c00, usermode=1) at >>> /usr/src/sys/amd64/amd64/trap.c:741 >>> #9 0xffffffff808d3386 in trap (frame=0xfffffe0101836c00) at >>> /usr/src/sys/amd64/amd64/trap.c:333 >>> #10 0xffffffff808b7af1 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 >> >> This tread is another program from the news system: >> 668 Thread 101245 (PID=49124: innfeed) sched_switch (td=0xfffff800b642aa00, >> newtd=0xfffff8000285f000, flags=) at >> /usr/src/sys/kern/sched_ule.c:1973 >> >>> >>> I strongly suspect that this is thread that we were looking for. >>> I think that it has the vnode lock in the shared mode while trying to fault in a >>> page. >>> --clip-- > > Okay. Luckily for us, it seems that 'm' is available in frame 5. It also > happens to be the first field of 'struct faultstate'. So, could you please go > to frame and print '*m' and '*(struct faultstate *)m' ? > (kgdb) fr 4 #4 0xffffffff8089d1c1 in vm_page_busy_sleep (m=0xfffff800df68cd40, wmesg=) at /usr/src/sys/vm/vm_page.c:753 753 msleep(m, vm_page_lockptr(m), PVM | PDROP, wmesg, 0); (kgdb) print *m $1 = {plinks = {q = {tqe_next = 0xfffff800dc5d85b0, tqe_prev = 0xfffff800debf3bd0}, s = {ss = {sle_next = 0xfffff800dc5d85b0}, pv = 0xfffff800debf3bd0}, memguard = {p = 18446735281313646000, v = 18446735281353604048}}, listq = {tqe_next = 0x0, tqe_prev = 0xfffff800dc5d85c0}, object = 0xfffff800b62e9c60, pindex = 11, phys_addr = 3389358080, md = {pv_list = { tqh_first = 0x0, tqh_last = 0xfffff800df68cd78}, pv_gen = 426, pat_mode = 6}, wire_count = 0, busy_lock = 6, hold_count = 0, flags = 0, aflags = 2 '\002', oflags = 0 '\0', queue = 0 '\0', psind = 0 '\0', segind = 3 '\003', order = 13 '\r', pool = 0 '\0', act_count = 0 '\0', valid = 0 '\0', dirty = 0 '\0'} (kgdb) print *(struct faultstate *)m $2 = {m = 0xfffff800dc5d85b0, object = 0xfffff800debf3bd0, pindex = 0, first_m = 0xfffff800dc5d85c0, first_object = 0xfffff800b62e9c60, first_pindex = 11, map = 0xca058000, entry = 0x0, lookup_still_valid = -546779784, vp = 0x6000001aa} (kgdb)